From patchwork Fri Apr 12 21:59:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084935 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="cMMj6xev"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJN5z4xz9s71 for ; Sat, 13 Apr 2019 08:00:04 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727157AbfDLWAD (ORCPT ); Fri, 12 Apr 2019 18:00:03 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:45036 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727156AbfDLWAD (ORCPT ); Fri, 12 Apr 2019 18:00:03 -0400 Received: by mail-wr1-f68.google.com with SMTP id y7so13695496wrn.11 for ; Fri, 12 Apr 2019 15:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wbnW/wYBeGzIP4i/h68N60AU3yOwxgPsQoXdv9s68Zc=; b=cMMj6xevEeuxBZj/8A5Dscp228TrbWGQUjJXVsozApd7l1N1LNCeEDDCg8c8KWHKk1 7THK7ULupH+gO7RDW/wpdenP5KdKs6Q0dnXQwX9FGcrQRELq9C86Ic2XdVMGs2aWK4MQ h9tOSCg33mEhrKUxS6nL1h8cd+mzqyR03Qwle6xmZYCtlRYzg4c7kLm0Rx8wl+OwxCKu xF13lI+dmKE7yZ2A/8egMlqHhk8cV0pKnbHGpx2W3mMG9NdlhUXIb+XRamp9dwWXfBkq CF168QCegv14jnWEfti7y0WRGsNig+Pys8m2pvHNSUmzbBpEm3cnCnjrehqZbHjrcjJv /nAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wbnW/wYBeGzIP4i/h68N60AU3yOwxgPsQoXdv9s68Zc=; b=BqY0fE0CEq7B6dW1WS8IJ3ajnJNVHyIJ7LcRB4woeDcz6k2S03xXwSDFroKU3sF9VM 84oj31ZVj5CvN884wjy7Wznflw4Jfs5sFDpUruK6If6dvEG78rJbrtMc+MltZSzoMVj7 DUyuun+ugF0EM0mWAOinyVHJl622RL6Dw20Hqzf/Hsp7bgoHC2uNOMP6vZ6mH7WJ16Al ER6/8I5eoJFbxZ93z/e1iQkyiVEaRfNIwavM6Akay7G6DjQGXyIIcCKNhP9eAO43VHTh 49TBQP8Lz3hAIjnTGaErvKVg2xkh8UInWLcVdbmDtCvtHtRkAEkhxVsuWBgmktNgbJBI yrRg== X-Gm-Message-State: APjAAAUF2c2np+HSJPzwAHETm8z51ItQJt3oi7LNVmogpPqi9xWsp0zX jXdW0SxQw2FAqLficS31E1f70A== X-Google-Smtp-Source: APXvYqxYLoQJKhWphQMtw444nwtwRECXpzxKObCwb0cYtptGlIPAxsdUVgJlKgGh6TrjEtKRznxaBA== X-Received: by 2002:adf:eb84:: with SMTP id t4mr10284618wrn.43.1555106401617; Fri, 12 Apr 2019 15:00:01 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:01 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 01/19] bpf: refactor propagate_liveness to eliminate duplicated for loop Date: Fri, 12 Apr 2019 22:59:34 +0100 Message-Id: <1555106392-20117-2-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Propagation for register and stack slot are finished in separate for loop, while they are perfect to be put into a single loop. This could also let them share some common variables in later patches. Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 15ab6fa..da285df 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6254,10 +6254,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, return err; } } - } - /* ... and stack slots */ - for (frame = 0; frame <= vstate->curframe; frame++) { + /* Propagate stack slots. */ state = vstate->frame[frame]; parent = vparent->frame[frame]; for (i = 0; i < state->allocated_stack / BPF_REG_SIZE && From patchwork Fri Apr 12 21:59:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084973 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="zT+VFuZG"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsKD52hYz9s71 for ; Sat, 13 Apr 2019 08:00:48 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727342AbfDLWAr (ORCPT ); Fri, 12 Apr 2019 18:00:47 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36326 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727167AbfDLWAE (ORCPT ); Fri, 12 Apr 2019 18:00:04 -0400 Received: by mail-wm1-f65.google.com with SMTP id h18so12655587wml.1 for ; Fri, 12 Apr 2019 15:00:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Sr2szAt1RChwWhkcjqTjVQ0F5Kfgn2Tb01LXULXSNXs=; b=zT+VFuZGzpFRdXp4loPUCo3/vN/OwoFG9d75xztI5MZWPCDBkThB4G4/JE6ossS0E+ OHEUO469mpWd8Zb4XDLk/FjxrhFPU6ZW3m1PN+/cr1jSVjpZ9cb5HiR/7aKcZBb9WiUO WN5vnrkg/zUsuG2ScyVWLWuKBw1+QlCBjvdlh0WSCgzaFPp07y8bcBPgQabguEhNsmft RxZM1U4CafU/Oq5EBzNTXwPDYmXWAXUNV0ohf+aXVWe9Kt6XMbuV2iyS52y1dxplUQR2 WXKmiC66lEx0p17v85lL6cLN5sym3MP/x3+ScrVM5mqZJNtrA1c2KHKZfyGUhw8cIpKy p2Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Sr2szAt1RChwWhkcjqTjVQ0F5Kfgn2Tb01LXULXSNXs=; b=M1xtpvSUO2j/66XEdtF+dwBcw6Sbw9PwN9zReA9OKlXj6dh9rSbyx+QATO+WO/+gOS v1Xkr14UIqpaVjcvJz8YePgTmTfqtZq62FvKylrbtEI17mfwTuaKLW3/QxpX0WRQwsIj YWhT1zeYUfHZmLT1VmEYWp5aY08+WahcBpzzC1TPuVGC31apxPjzInOo7LMk99CiVhwH lM7uX87e+Rqzsy/WNuII9UkwHQTeYlVwArC7lEPG4Z8cQrvk6ORP1IaCJN6Y6GQIySQP F+fk+X4PqAxxONeWd556sn8m4Esb3WFUA7CscQBmMt9rNbNKB3FxhyzKwGvkStpxFiKT yjZw== X-Gm-Message-State: APjAAAWt8uzoBaTtsAzJQntMp2UyAmeHcsC9nPWJp1+zBSPpw1S9wjDW 0LRSOoSBK/XJxxoL5YuRicQFNg== X-Google-Smtp-Source: APXvYqzw4CptVkem1bJAxeLBTrwhPVTlIWotzC6fWP4j4DiArrqQW7ljm7ogY5ta4AHmpK5/MdsS4g== X-Received: by 2002:a05:600c:219:: with SMTP id 25mr13330385wmi.7.1555106402801; Fri, 12 Apr 2019 15:00:02 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:02 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 02/19] bpf: refactor propagate_liveness to eliminate code redundance Date: Fri, 12 Apr 2019 22:59:35 +0100 Message-Id: <1555106392-20117-3-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Access to reg states were not factored out, the consequence is long code for dereferencing them which made the indentation not good for reading. This patch factor out these code so the core code in the loop could be easier to follow. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index da285df..6fd36a8 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6232,8 +6232,9 @@ static int propagate_liveness(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate, struct bpf_verifier_state *vparent) { - int i, frame, err = 0; + struct bpf_reg_state *state_reg, *parent_reg; struct bpf_func_state *state, *parent; + int i, frame, err = 0; if (vparent->curframe != vstate->curframe) { WARN(1, "propagate_live: parent frame %d current frame %d\n", @@ -6243,28 +6244,33 @@ static int propagate_liveness(struct bpf_verifier_env *env, /* Propagate read liveness of registers... */ BUILD_BUG_ON(BPF_REG_FP + 1 != MAX_BPF_REG); for (frame = 0; frame <= vstate->curframe; frame++) { + parent = vparent->frame[frame]; + state = vstate->frame[frame]; + parent_reg = parent->regs; + state_reg = state->regs; /* We don't need to worry about FP liveness, it's read-only */ for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { - if (vparent->frame[frame]->regs[i].live & REG_LIVE_READ) + if (parent_reg[i].live & REG_LIVE_READ) continue; - if (vstate->frame[frame]->regs[i].live & REG_LIVE_READ) { - err = mark_reg_read(env, &vstate->frame[frame]->regs[i], - &vparent->frame[frame]->regs[i]); - if (err) - return err; - } + if (!(state_reg[i].live & REG_LIVE_READ)) + continue; + err = mark_reg_read(env, &state_reg[i], &parent_reg[i]); + if (err) + return err; } /* Propagate stack slots. */ - state = vstate->frame[frame]; - parent = vparent->frame[frame]; for (i = 0; i < state->allocated_stack / BPF_REG_SIZE && i < parent->allocated_stack / BPF_REG_SIZE; i++) { - if (parent->stack[i].spilled_ptr.live & REG_LIVE_READ) + parent_reg = &parent->stack[i].spilled_ptr; + state_reg = &state->stack[i].spilled_ptr; + if (parent_reg->live & REG_LIVE_READ) continue; - if (state->stack[i].spilled_ptr.live & REG_LIVE_READ) - mark_reg_read(env, &state->stack[i].spilled_ptr, - &parent->stack[i].spilled_ptr); + if (!(state_reg->live & REG_LIVE_READ)) + continue; + err = mark_reg_read(env, state_reg, parent_reg); + if (err) + return err; } } return err; From patchwork Fri Apr 12 21:59:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084938 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="b8rbV+5w"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJS4TDNz9s71 for ; Sat, 13 Apr 2019 08:00:08 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727197AbfDLWAG (ORCPT ); Fri, 12 Apr 2019 18:00:06 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41339 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727177AbfDLWAF (ORCPT ); Fri, 12 Apr 2019 18:00:05 -0400 Received: by mail-wr1-f68.google.com with SMTP id r4so13684395wrq.8 for ; Fri, 12 Apr 2019 15:00:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=I1cowfeKkPG5Vz6uYfZSI/ZUZumixhNFe5aVN4gWJYo=; b=b8rbV+5wrPSH+WunviEuLpMttCAO3JJ42jbNLk2BiFwuV0dAE9a8qB5FOPw+pDU6xb ETtQ5tXhOCfgevYS1HwyhY/dDrx47hqAN9E/P/HXp3fsA7tPHmAMDokAj7RlimewoZt+ Y561e7b7QkaMKZO8Hcy9y3IDNDFFpi/tV5uPSNGeI7JeYMCV5R/FCbWidyIw0oeRdgdG hkOevROZnNflpiYR7uAqrbgX8kOl6Xz4VgxKVZvs1pZJrQgjIBuetGIee0ooPHBGklsh IPOyyQDgopnM7m2zLil4y1ZYYTbQ7O8qg1GKjDjYq+WSvM39JZ+vnG6X3wMrBiAEVWdo qwVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=I1cowfeKkPG5Vz6uYfZSI/ZUZumixhNFe5aVN4gWJYo=; b=h7HU4U5xr0fyXazXg4jqeQU1ax+H3REof48ICmjWPHqy3WDgly656XuzTBzDePW/Xs yclkGTedx+NrfRJr+41fFbCwnyamHMPL5i6kXR2ROgjLkz8RtiXjw/oh5kA+HVXVTSfn 5qvdGV7OYIYC2hTByaH++w/yYNxnxfnEIQ8z82yr4sU4BXinOMxCJ991GVLpHrYn6Ov3 QBgTh/MxGOCmp4s+BCUBmy7dHHC1Phw53nBqvy5EL4LCF4LIZGLuWrhQoLiGKir6ohjv bCOT20GksDoME6H0QqiD2OyxzMev9FKskY550PvBsGXNz60gz0LqGsm70dKz8zKKjW+5 sYmw== X-Gm-Message-State: APjAAAW6FkNhIHd/n5RVbKKuok0aUObWlkfpxCOowpfmKjUWxS6gBNgy 0fbzvBFeWzbaeI0oM4CHSXFFbA== X-Google-Smtp-Source: APXvYqyP010ROPGTgcE7El+BPG/GWuQpn3ZYkwsSl1k5ilyXneiyQ3zF95hMeelMxKCD8DWp3cva/Q== X-Received: by 2002:adf:cf0e:: with SMTP id o14mr36176637wrj.182.1555106404023; Fri, 12 Apr 2019 15:00:04 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:03 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 03/19] bpf: factor out reg and stack slot propagation into "propagate_liveness_reg" Date: Fri, 12 Apr 2019 22:59:36 +0100 Message-Id: <1555106392-20117-4-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org After code refactor in previous patches, the propagation logic inside the for loop in "propagate_liveness" becomes clear that they are good enough to be factored out into a common function "propagate_liveness_reg". Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6fd36a8..3fdb301 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6221,6 +6221,22 @@ static bool states_equal(struct bpf_verifier_env *env, return true; } +static int propagate_liveness_reg(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, + struct bpf_reg_state *parent_reg) +{ + int err; + + if (parent_reg->live & REG_LIVE_READ || !(reg->live & REG_LIVE_READ)) + return 0; + + err = mark_reg_read(env, reg, parent_reg); + if (err) + return err; + + return 0; +} + /* A write screens off any subsequent reads; but write marks come from the * straight-line code between a state and its parent. When we arrive at an * equivalent state (jump target or such) we didn't arrive by the straight-line @@ -6250,11 +6266,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, state_reg = state->regs; /* We don't need to worry about FP liveness, it's read-only */ for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { - if (parent_reg[i].live & REG_LIVE_READ) - continue; - if (!(state_reg[i].live & REG_LIVE_READ)) - continue; - err = mark_reg_read(env, &state_reg[i], &parent_reg[i]); + err = propagate_liveness_reg(env, &state_reg[i], + &parent_reg[i]); if (err) return err; } @@ -6264,11 +6277,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, i < parent->allocated_stack / BPF_REG_SIZE; i++) { parent_reg = &parent->stack[i].spilled_ptr; state_reg = &state->stack[i].spilled_ptr; - if (parent_reg->live & REG_LIVE_READ) - continue; - if (!(state_reg->live & REG_LIVE_READ)) - continue; - err = mark_reg_read(env, state_reg, parent_reg); + err = propagate_liveness_reg(env, state_reg, + parent_reg); if (err) return err; } From patchwork Fri Apr 12 21:59:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084939 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="wAcwXDVb"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJT6Yp7z9s71 for ; Sat, 13 Apr 2019 08:00:09 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727204AbfDLWAI (ORCPT ); Fri, 12 Apr 2019 18:00:08 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:37878 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727198AbfDLWAI (ORCPT ); Fri, 12 Apr 2019 18:00:08 -0400 Received: by mail-wr1-f65.google.com with SMTP id w10so13752369wrm.4 for ; Fri, 12 Apr 2019 15:00:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5JznXDOEWc1oAs2pt0jdnGJ632ezz+lDAgpcI6vofg8=; b=wAcwXDVb3uA1GTO8+OGOQwPdzAtJN9n2KBmy2ThZMxiCcGEjOWvISHJfk5VxDejhXD hOgzLxiUtLwwONBXDtkX3zGKsNbRdIrD8lsNGSVq6seJ24ODXl1gLPR9WsIGVoJgSJVX Vmg8Te0Vu9BfdeRiOybWhWgku9f2rPzNULPjXclt3eDEdWTu1fPoNeP247YxUPeVFWFq ypT7+iVxDktIFVLGZG7k7GdzqwRssoyFF2Fy3hd6qki9eAWwu1p4/LU2b6EfPzWiXq+N ip9lxfqwW32Y98iriaazr97wWDoSx6vojBgrZiDHWFyneG/Etwu7xloWI+QfWMw89r3Y +Lcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5JznXDOEWc1oAs2pt0jdnGJ632ezz+lDAgpcI6vofg8=; b=Gj/k2OdYcQ8eKzsIJoGCZtPaK3DCjKHarwUT9mW7zamwx3lTjX9gJ1TfsYcrNaLVbU oC66DhyL6JJfU6ICgElRQ861TRrIkvHOObjM6S6PaogKTjqEM5pTunnbwD8fOgS2Y7bx 3HcdRjk+mB9cq4wBs2w1t/1TLczj6b+dZZzjkGVcz+sswUjw7SBvYQ6GJZNdvyCUEjX2 5696kosL6IEdp6t1jMR/mPD2dKML/YBRXpmUtFo5qFYW3o3gizT+L6QpYpn87R6IEP1Q /r1agfPCQ8jk0XRylt+uZvpMnhd25bZlwKx3Gbrgs1919qKomTo240VKOg5lWJbGSh8E JC8w== X-Gm-Message-State: APjAAAU8lHIf/HIMd/bSimiogVLbM2UXR5gEgXGp/5sGm9sbjNJjIqWc ZMoH7RlNycp9vtNAcI33h+Felg== X-Google-Smtp-Source: APXvYqxpNfHFHCxRaKYjKy+uisRIx+lSFnNBd1YHd1K6KRtqnNB8IdNhQu1bkTs+uS4+hF17JXrMRw== X-Received: by 2002:a5d:4492:: with SMTP id j18mr38452572wrq.212.1555106406219; Fri, 12 Apr 2019 15:00:06 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:04 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 04/19] bpf: refactor "check_reg_arg" to eliminate code redundancy Date: Fri, 12 Apr 2019 22:59:37 +0100 Message-Id: <1555106392-20117-5-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org There are a few "regs[regno]" here are there across "check_reg_arg", this patch factor it out into a simple "reg" pointer. The intention is to simplify code indentation and make the later patches in this set look cleaner. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3fdb301..c722015 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1177,30 +1177,32 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; - struct bpf_reg_state *regs = state->regs; + struct bpf_reg_state *reg, *regs = state->regs; if (regno >= MAX_BPF_REG) { verbose(env, "R%d is invalid\n", regno); return -EINVAL; } + reg = ®s[regno]; if (t == SRC_OP) { /* check whether register used as source operand can be read */ - if (regs[regno].type == NOT_INIT) { + if (reg->type == NOT_INIT) { verbose(env, "R%d !read_ok\n", regno); return -EACCES; } /* We don't need to worry about FP liveness because it's read-only */ - if (regno != BPF_REG_FP) - return mark_reg_read(env, ®s[regno], - regs[regno].parent); + if (regno == BPF_REG_FP) + return 0; + + return mark_reg_read(env, reg, reg->parent); } else { /* check whether register used as dest operand can be written to */ if (regno == BPF_REG_FP) { verbose(env, "frame pointer is read only\n"); return -EACCES; } - regs[regno].live |= REG_LIVE_WRITTEN; + reg->live |= REG_LIVE_WRITTEN; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } From patchwork Fri Apr 12 21:59:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084941 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="X7hc/3Gk"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJX05f9z9s71 for ; Sat, 13 Apr 2019 08:00:12 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727223AbfDLWAK (ORCPT ); Fri, 12 Apr 2019 18:00:10 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:37953 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727201AbfDLWAJ (ORCPT ); Fri, 12 Apr 2019 18:00:09 -0400 Received: by mail-wr1-f66.google.com with SMTP id k11so13745828wro.5 for ; Fri, 12 Apr 2019 15:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0Bth1/39lYXfmckbmsiuKqNidJbiikbB2x6UCWPFtnE=; b=X7hc/3GkhiE2EOcLELr74UaCPhQ5nCRpptUAks8871eObf54Y4+G/SrERs1q0NEU0D ssSibwaCGLdCBNOGLiwUVrub655Lj50wXi7cO6q6PlQjdETwn/juLkAELh9P6Xqbeo5f qhewwAyF6jDwlEzKMIKT2PzHawTDlgVF1Vg1f6qbdIte74RCF9h5qKhIqQkhvNO+eNRP MnwwmBVSm/91wi16EUDSMzZ91GrgYfAmxhBE3pmxEACvp8riCz9FJ7t/aK1cOe7js3wl 6z6oxIG7sHWvaDYhT7uO+2/b+PmxnbLRb9WCqPQc4p53bmEF2TzHy+AziOkpRuTHL79k Xy7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0Bth1/39lYXfmckbmsiuKqNidJbiikbB2x6UCWPFtnE=; b=GFI16yShzKOS4HoMig2Yphqmn8/KwXeuSLPFmlxMcIHUnA/uy7hTDpHWFH+C6VrVSN 4QChQPzrgVmM6w30V7iSNgFujA3QselLoaNNKKeQlkuXLrCMKuFtmVWrmvhSOrPjKO65 Zjxi8d3w/HJb2NZE4HGek5mxk7c+HvPFi80MKXQ/PnQFF9ktcTkOKgiuZM8s2UnwAdC4 EQDnQIbU8W7bikZJem6Gk6BjDyrUFi5GasSIDNibRqYdE6kewBKTLnDZ1ZlVgrQb0eCs nqAennqtXtHJX3m62eE4irsfyXxoiIBwVZKKRmJ7L2LIROsl2w04HVDsyThffVggzvKl bxIw== X-Gm-Message-State: APjAAAVvsg9QY5Hoz/UgLk5cgQoKeqZEziId9xDdlxD3nRl1Pw3wYctw HCgVURvzaHFnLtRi0TUVm0p0O6Moahk= X-Google-Smtp-Source: APXvYqyfYVqH5BF4Cv1X5yIPKccmmEYQ967wfBjuRLAl2H8J8kPitchXYk+vrvTL6b5G9LvRSNdY/A== X-Received: by 2002:adf:ce8c:: with SMTP id r12mr18363426wrn.60.1555106407407; Fri, 12 Apr 2019 15:00:07 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:06 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 05/19] bpf: split read liveness into REG_LIVE_READ64 and REG_LIVE_READ32 Date: Fri, 12 Apr 2019 22:59:38 +0100 Message-Id: <1555106392-20117-6-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Register liveness infrastructure doesn't track register read width at the moment, while the width information will be needed for the later 32-bit safety analysis pass. This patch take the first step to split read liveness into REG_LIVE_READ64 and REG_LIVE_READ32. Liveness propagation code are updated accordingly. They are taught to understand how to propagate REG_LIVE_READ64 and REG_LIVE_READ32 at the same propagation iteration. For example, "mark_reg_read" now propagate "flags" which could be multiple read bits instead of the single REG_LIVE_READ64. A write still screen off all width of reads. Signed-off-by: Jiong Wang --- include/linux/bpf_verifier.h | 8 +-- kernel/bpf/verifier.c | 119 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 113 insertions(+), 14 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index b3ab61f..fba0ebb 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -36,9 +36,11 @@ */ enum bpf_reg_liveness { REG_LIVE_NONE = 0, /* reg hasn't been read or written this branch */ - REG_LIVE_READ, /* reg was read, so we're sensitive to initial value */ - REG_LIVE_WRITTEN, /* reg was written first, screening off later reads */ - REG_LIVE_DONE = 4, /* liveness won't be updating this register anymore */ + REG_LIVE_READ32 = 0x1, /* reg was read, so we're sensitive to initial value */ + REG_LIVE_READ64 = 0x2, /* likewise, but full 64-bit content matters */ + REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64, + REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */ + REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ }; struct bpf_reg_state { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index c722015..3c5ca00 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1135,7 +1135,7 @@ static int check_subprogs(struct bpf_verifier_env *env) */ static int mark_reg_read(struct bpf_verifier_env *env, const struct bpf_reg_state *state, - struct bpf_reg_state *parent) + struct bpf_reg_state *parent, u8 flags) { bool writes = parent == state->parent; /* Observe write marks */ int cnt = 0; @@ -1150,17 +1150,17 @@ static int mark_reg_read(struct bpf_verifier_env *env, parent->var_off.value, parent->off); return -EFAULT; } - if (parent->live & REG_LIVE_READ) + if ((parent->live & REG_LIVE_READ) == flags) /* The parentage chain never changes and - * this parent was already marked as LIVE_READ. + * this parent was already marked with all read bits. * There is no need to keep walking the chain again and - * keep re-marking all parents as LIVE_READ. + * keep re-marking all parents with reads bits in flags. * This case happens when the same register is read * multiple times without writes into it in-between. */ break; /* ... then we depend on parent's value */ - parent->live |= REG_LIVE_READ; + parent->live |= flags; state = parent; parent = state->parent; writes = true; @@ -1172,12 +1172,95 @@ static int mark_reg_read(struct bpf_verifier_env *env, return 0; } +/* This function is supposed to be used by the following 32-bit optimization + * code only. It returns TRUE if the source or destination register operates + * on 64-bit, otherwise return FALSE. + */ +static bool is_reg64(struct bpf_insn *insn, u32 regno, + struct bpf_reg_state *reg, enum reg_arg_type t) +{ + u8 code, class, op; + + code = insn->code; + class = BPF_CLASS(code); + op = BPF_OP(code); + if (class == BPF_JMP) { + /* BPF_EXIT will reach here because of return value readability + * test for "main" which has s32 return value. + */ + if (op == BPF_EXIT) + return false; + if (op == BPF_CALL) { + /* BPF to BPF call will reach here because of marking + * caller saved clobber with DST_OP_NO_MARK for which we + * don't care the register def because they are anyway + * marked as NOT_INIT already. + */ + if (insn->src_reg == BPF_PSEUDO_CALL) + return false; + /* Helper call will reach here because of arg type + * check. Conservatively marking all args as 64-bit. + */ + return true; + } + } + + if (class == BPF_ALU64 || class == BPF_JMP || + /* BPF_END always use BPF_ALU class. */ + (class == BPF_ALU && op == BPF_END && insn->imm == 64)) + return true; + + if (class == BPF_ALU || class == BPF_JMP32) + return false; + + if (class == BPF_LDX) { + if (t != SRC_OP) + return BPF_SIZE(code) == BPF_DW; + /* LDX source must be ptr. */ + return true; + } + + if (class == BPF_STX) { + if (reg->type != SCALAR_VALUE) + return true; + return BPF_SIZE(code) == BPF_DW; + } + + if (class == BPF_LD) { + u8 mode = BPF_MODE(code); + + /* LD_IMM64 */ + if (mode == BPF_IMM) + return true; + + /* Both LD_IND and LD_ABS return 32-bit data. */ + if (t != SRC_OP) + return false; + + /* Implicit ctx ptr. */ + if (regno == BPF_REG_6) + return true; + + /* Explicit source could be any width. */ + return true; + } + + if (class == BPF_ST) + /* The only source register for BPF_ST is a ptr. */ + return true; + + /* Conservatively return true at default. */ + return true; +} + static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, enum reg_arg_type t) { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; + struct bpf_insn *insn = env->prog->insnsi + env->insn_idx; struct bpf_reg_state *reg, *regs = state->regs; + bool rw64; if (regno >= MAX_BPF_REG) { verbose(env, "R%d is invalid\n", regno); @@ -1185,6 +1268,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, } reg = ®s[regno]; + rw64 = is_reg64(insn, regno, reg, t); if (t == SRC_OP) { /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { @@ -1195,7 +1279,8 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, if (regno == BPF_REG_FP) return 0; - return mark_reg_read(env, reg, reg->parent); + return mark_reg_read(env, reg, reg->parent, + rw64 ? REG_LIVE_READ64 : REG_LIVE_READ32); } else { /* check whether register used as dest operand can be written to */ if (regno == BPF_REG_FP) { @@ -1382,7 +1467,8 @@ static int check_stack_read(struct bpf_verifier_env *env, state->regs[value_regno].live |= REG_LIVE_WRITTEN; } mark_reg_read(env, ®_state->stack[spi].spilled_ptr, - reg_state->stack[spi].spilled_ptr.parent); + reg_state->stack[spi].spilled_ptr.parent, + REG_LIVE_READ64); return 0; } else { int zeros = 0; @@ -1399,7 +1485,9 @@ static int check_stack_read(struct bpf_verifier_env *env, return -EACCES; } mark_reg_read(env, ®_state->stack[spi].spilled_ptr, - reg_state->stack[spi].spilled_ptr.parent); + reg_state->stack[spi].spilled_ptr.parent, + size == BPF_REG_SIZE + ? REG_LIVE_READ64 : REG_LIVE_READ32); if (value_regno >= 0) { if (zeros == size) { /* any size read into register is zero extended, @@ -2337,7 +2425,9 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, * the whole slot to be marked as 'read' */ mark_reg_read(env, &state->stack[spi].spilled_ptr, - state->stack[spi].spilled_ptr.parent); + state->stack[spi].spilled_ptr.parent, + access_size == BPF_REG_SIZE + ? REG_LIVE_READ64 : REG_LIVE_READ32); } return update_stack_depth(env, state, min_off); } @@ -6227,12 +6317,19 @@ static int propagate_liveness_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, struct bpf_reg_state *parent_reg) { + u8 parent_bits = parent_reg->live & REG_LIVE_READ; + u8 bits = reg->live & REG_LIVE_READ; + u8 bits_diff = parent_bits ^ bits; + u8 bits_prop = bits_diff & bits; int err; - if (parent_reg->live & REG_LIVE_READ || !(reg->live & REG_LIVE_READ)) + /* "reg" and "parent_reg" has the same read bits, or the bit doesn't + * belong to "reg". + */ + if (!bits_diff || !bits_prop) return 0; - err = mark_reg_read(env, reg, parent_reg); + err = mark_reg_read(env, reg, parent_reg, bits_prop); if (err) return err; From patchwork Fri Apr 12 21:59:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084942 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="qLrve8N9"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJZ3ClKz9sB8 for ; Sat, 13 Apr 2019 08:00:14 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727242AbfDLWAN (ORCPT ); Fri, 12 Apr 2019 18:00:13 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:37956 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727209AbfDLWAK (ORCPT ); Fri, 12 Apr 2019 18:00:10 -0400 Received: by mail-wr1-f67.google.com with SMTP id k11so13745893wro.5 for ; Fri, 12 Apr 2019 15:00:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=M4oS4lAyke3Kgzi/tLbCTTag+k010jqZDCyj0xY9SkE=; b=qLrve8N9AUJ4DxMgFP2HyMFx49nawGM6lwp+1vWYTFEh24UXSqbdUsaNxnCVarGSCF bBAHkY+KUTAK3WTZkgp6gn7qt5oHirKHCPJg8EXa/fEZR/Y8mB/+5rmT9XneSDAQtF4e N8qBmvoFgsnU72ChlfaYePQgBbcj46PgNN6eYIHzH3kUta5OH3zsf6OBSzrhQWbkI6Et yMKO1vYUsPatTHMhzhWCWXeNQXCxT6IkBxv0vH1YSPS3OwkB6TJPll2X9rjH3V7xTEcr gHjppeuXQknPgGG3ojAdyWzY6xaCgB0BV44/zDC0VmuijDCZC/lFKRkAAH1jAGxeu/Q8 yBxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=M4oS4lAyke3Kgzi/tLbCTTag+k010jqZDCyj0xY9SkE=; b=eM/j0WNqXHeR8uGNdth5rN6aiV2+94WR3MBYUIgfUtrrYHjnT9/FPqKYV2EIl26wGF aBCw2qfI+ipBNDGXA06X6qS3mEJcTi6Fh2G8bePhclLtC+FpzJAin9t+yc6Xe0ZIBe0p Yo3Z+ulKLKuzvHZ5QRUUHfBqk2A5lLOJTfEyBMldB7Wrv0LSQdlqfc/2aT9xn92iLV+x jgEuP9aL0BlbalitaiIdVjQmNW12mRveCyrCffYrSAcUXlJUZOhgDj2A+33ChZYUCcZ1 6IxIT2UygFlg/PKhA26/7MVQMp7wC2H32TxV+ub0appWq0oCEb1q038gnPSFiz/q93iu /TOw== X-Gm-Message-State: APjAAAWMvKcsKjiU3cwLpyYWoQ8BrkqPNutAFS2r6O1Y6iTLjHVPWLRk aGlVz/FhO/Xri9cqkvc9hQcdOQ== X-Google-Smtp-Source: APXvYqwlEXG535b8Lq6fnGjlQ3jQBUXUai+DXcZ8SLPG5w2T3m5lucKqxIvF6+HeQ3zFkfqUlxwClA== X-Received: by 2002:adf:fd41:: with SMTP id h1mr34930069wrs.155.1555106408601; Fri, 12 Apr 2019 15:00:08 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:07 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 06/19] bpf: mark lo32 writes that should be zero extended into hi32 Date: Fri, 12 Apr 2019 22:59:39 +0100 Message-Id: <1555106392-20117-7-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org eBPF ISA specification requires high 32-bit cleared when low 32-bit sub-register is written. This applies to destination register of ALU32 etc. JIT back-ends must guarantee this semantic when doing code-gen. x86-64 and arm64 ISA has the same semantic, so the corresponding JIT back-end doesn't need to do extra work. However, 32-bit arches (arm, nfp etc.) and some other 64-bit arches (powerpc, sparc etc), need explicit zero extension sequence to meet such semantic. This is important, because for code the following: u64_value = (u64) u32_value ... other uses of u64_value compiler could exploit the semantic described above and save those zero extensions for extending u32_value to u64_value. Hardware, runtime, or BPF JIT back-ends, are responsible for guaranteeing this. Some benchmarks show ~40% sub-register writes out of total insns, meaning ~40% extra code-gen ( could go up to more for some arches which requires two shifts for zero extension) because JIT back-end needs to do extra code-gen for all such instructions. However this is not always necessary in case u32_value is never cast into a u64, which is quite normal in real life program. So, it would be really good if we could identify those places where such type cast happened, and only do zero extensions for them, not for the others. This could save a lot of BPF code-gen. Algo: - Record indices of instructions that do sub-register def (write). And these indices need to stay with reg state so path pruning and bpf to bpf function call could be handled properly. These indices are kept up to date while doing insn walk. - A full register read on an active sub-register def marks the def insn as needing zero extension on dst register. - A new sub-register write overrides the old one. A new full register write makes the register free of zero extension on dst register. - When propagating read64 during path pruning, also marks def insns whose defs are hanging active sub-register. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf_verifier.h | 6 ++++++ kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index fba0ebb..c1923a5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -133,6 +133,11 @@ struct bpf_reg_state { * pointing to bpf_func_state. */ u32 frameno; + /* Tracks subreg definition. The stored value is the insn_idx of the + * writing insn. This is safe because subreg_def is used before any insn + * patching which only happens after main verification finished. + */ + s32 subreg_def; enum bpf_reg_liveness live; }; @@ -234,6 +239,7 @@ struct bpf_insn_aux_data { int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ int sanitize_stack_off; /* stack slot to be cleared */ bool seen; /* this insn was processed by the verifier */ + bool zext_dst; /* this insn zero extend dst reg */ u8 alu_state; /* used in combination with alu_limit */ unsigned int orig_idx; /* original instruction index */ }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3c5ca00..3bcbd9b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -980,6 +980,7 @@ static void mark_reg_not_init(struct bpf_verifier_env *env, __mark_reg_not_init(regs + regno); } +#define DEF_NOT_SUBREG (-1) static void init_reg_state(struct bpf_verifier_env *env, struct bpf_func_state *state) { @@ -990,6 +991,7 @@ static void init_reg_state(struct bpf_verifier_env *env, mark_reg_not_init(env, regs, i); regs[i].live = REG_LIVE_NONE; regs[i].parent = NULL; + regs[i].subreg_def = DEF_NOT_SUBREG; } /* frame pointer */ @@ -1253,6 +1255,19 @@ static bool is_reg64(struct bpf_insn *insn, u32 regno, return true; } +static void mark_insn_zext(struct bpf_verifier_env *env, + struct bpf_reg_state *reg) +{ + s32 def_idx = reg->subreg_def; + + if (def_idx == DEF_NOT_SUBREG) + return; + + env->insn_aux_data[def_idx].zext_dst = true; + /* The dst will be zero extended, so won't be sub-register anymore. */ + reg->subreg_def = DEF_NOT_SUBREG; +} + static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, enum reg_arg_type t) { @@ -1279,6 +1294,9 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, if (regno == BPF_REG_FP) return 0; + if (rw64) + mark_insn_zext(env, reg); + return mark_reg_read(env, reg, reg->parent, rw64 ? REG_LIVE_READ64 : REG_LIVE_READ32); } else { @@ -1288,6 +1306,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, return -EACCES; } reg->live |= REG_LIVE_WRITTEN; + reg->subreg_def = rw64 ? DEF_NOT_SUBREG : env->insn_idx; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } @@ -2170,6 +2189,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn value_regno); if (reg_type_may_be_null(reg_type)) regs[value_regno].id = ++env->id_gen; + /* A load of ctx field could have different + * actual load size with the one encoded in the + * insn. When the dst is PTR, it is for sure not + * a sub-register. + */ + regs[value_regno].subreg_def = DEF_NOT_SUBREG; } regs[value_regno].type = reg_type; } @@ -3370,6 +3395,9 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); } + /* helper call must return full 64-bit R0. */ + regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; + /* update return register (already marked as written above) */ if (fn->ret_type == RET_INTEGER) { /* sets type to SCALAR_VALUE */ @@ -4301,6 +4329,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) */ *dst_reg = *src_reg; dst_reg->live |= REG_LIVE_WRITTEN; + dst_reg->subreg_def = DEF_NOT_SUBREG; } else { /* R1 = (u32) R2 */ if (is_pointer_value(env, insn->src_reg)) { @@ -4311,6 +4340,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) } else if (src_reg->type == SCALAR_VALUE) { *dst_reg = *src_reg; dst_reg->live |= REG_LIVE_WRITTEN; + dst_reg->subreg_def = env->insn_idx; } else { mark_reg_unknown(env, regs, insn->dst_reg); @@ -5374,6 +5404,8 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * Already marked as written above. */ mark_reg_unknown(env, regs, BPF_REG_0); + /* ld_abs load up to 32-bit skb data. */ + regs[BPF_REG_0].subreg_def = env->insn_idx; return 0; } @@ -6313,6 +6345,9 @@ static bool states_equal(struct bpf_verifier_env *env, return true; } +/* Return 0 if no propagation happened. Return negative error code if error + * happened. Otherwise, return the propagated bits. + */ static int propagate_liveness_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, struct bpf_reg_state *parent_reg) @@ -6333,7 +6368,7 @@ static int propagate_liveness_reg(struct bpf_verifier_env *env, if (err) return err; - return 0; + return bits_prop; } /* A write screens off any subsequent reads; but write marks come from the @@ -6367,8 +6402,10 @@ static int propagate_liveness(struct bpf_verifier_env *env, for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { err = propagate_liveness_reg(env, &state_reg[i], &parent_reg[i]); - if (err) + if (err < 0) return err; + if (err & REG_LIVE_READ64) + mark_insn_zext(env, &parent_reg[i]); } /* Propagate stack slots. */ @@ -6378,11 +6415,11 @@ static int propagate_liveness(struct bpf_verifier_env *env, state_reg = &state->stack[i].spilled_ptr; err = propagate_liveness_reg(env, state_reg, parent_reg); - if (err) + if (err < 0) return err; } } - return err; + return 0; } static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) From patchwork Fri Apr 12 21:59:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084944 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="GWmFLhj4"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJd0jNRz9sN4 for ; Sat, 13 Apr 2019 08:00:17 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727253AbfDLWAP (ORCPT ); Fri, 12 Apr 2019 18:00:15 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:40023 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727226AbfDLWAM (ORCPT ); Fri, 12 Apr 2019 18:00:12 -0400 Received: by mail-wm1-f67.google.com with SMTP id z24so12734204wmi.5 for ; Fri, 12 Apr 2019 15:00:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lbG4IZ70X0LLzHsqgrSvIlZXhNJfGsbmDOFJF5Yt8us=; b=GWmFLhj47R4tm6VuIQuYcTSjVTg2jfy3X4y6L+cQmBeKmoi6bx88LMk9W6aKvai9BB VyyzS266nDl/yAVaAMDtpx3N50qF5ek1ax5Hp8Ee5TLRGG6YPrLD6573K1b+0k8K+N3a yC3YAX66k6460y07ksCBqYD80MVN4oKJzSUQn0nGby8ghVGe0jlAdO61bx3ca6J0v1TI FBGCy4J6pdMaJeaoIJbkCTN8qL5Ywj4X1iIqUGWumfFrpXikqoaupbFlRCGadxFaOxav 9u+/VKQVk7dLIkhrLhCwek1aEaAJsytwfDDRW5b5RGxCo38jQxO7i/mJ+eE149mnt8ul jEng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lbG4IZ70X0LLzHsqgrSvIlZXhNJfGsbmDOFJF5Yt8us=; b=WDMWvqiAo5WfT8/gKTVBpezE61uNcMvbfWzsFXq+71RFJEOG918Z+NXpqwZXMuyk/R b+jDuGbYiV2OaB+Unk2sibr49solZ8uK2i03mT4EjfyAIiodKjuiCifdTof658zm+V6G Q9v7mUZdtWr36hj2/4VmWkus6B1Kfl17vJTw+NL3pgHCDzovUa8qC1aFdnug+tgMELlp l2Lt4Fsh1d/K2de6Pl6dcMss3KJozPo15AGwjq9mJ/cKYLgnmQM7RgoHo5JbLoSe+rfs sUNFA7rwPbP+YY+ZL1vyfTeqjEtUtfoEB6dHXnqI275iuWmeVYDZjo2wGLAxlPaWTJlz CK6Q== X-Gm-Message-State: APjAAAVmjbnUshUgFaUh4/721+vbnKlU0Rsko2G9eYOTWsTqt9gyPdDF a0BvyIyy4xBsE5q/GCIDJO0V1w== X-Google-Smtp-Source: APXvYqzZCmlsz7s1JH7zrdjFABsDPfz6j1TNqYDJkbEDLeFlbZ+l6kGqV6kfolZH0Gach5GCGFdsrg== X-Received: by 2002:a7b:c257:: with SMTP id b23mr12349019wmj.83.1555106409892; Fri, 12 Apr 2019 15:00:09 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:09 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 07/19] bpf: reduce false alarm by refining helper call arg types Date: Fri, 12 Apr 2019 22:59:40 +0100 Message-Id: <1555106392-20117-8-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Unlike BPF to BPF function call, BPF helper call is calls to native insns that verifier can't walk. So, verifier needs helper proto type descriptions for data-flow purpose. There is such information already, but it is not differentiate sub-register read with full register read. This patch split "enum bpf_arg_type" for sub-register read, and updated descriptions for several functions that shown frequent usage in one Cilium benchmark. "is_reg64" then taught about these new arg types. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf.h | 3 +++ kernel/bpf/core.c | 2 +- kernel/bpf/helpers.c | 2 +- kernel/bpf/verifier.c | 61 +++++++++++++++++++++++++++++++++++++++++++-------- net/core/filter.c | 28 +++++++++++------------ 5 files changed, 71 insertions(+), 25 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f15432d..884b8e1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -197,9 +197,12 @@ enum bpf_arg_type { ARG_CONST_SIZE, /* number of bytes accessed from memory */ ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ + ARG_CONST_SIZE32, /* Likewise, but size fits into 32-bit */ + ARG_CONST_SIZE32_OR_ZERO, /* Ditto */ ARG_PTR_TO_CTX, /* pointer to context */ ARG_ANYTHING, /* any (initialized) argument is ok */ + ARG_ANYTHING32, /* Likewise, but it is a 32-bit argument */ ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */ ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */ ARG_PTR_TO_INT, /* pointer to int */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ace8c22..2792eda 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2067,7 +2067,7 @@ const struct bpf_func_proto bpf_tail_call_proto = { .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_CONST_MAP_PTR, - .arg3_type = ARG_ANYTHING, + .arg3_type = ARG_ANYTHING32, }; /* Stub for JITs that only support cBPF. eBPF programs are interpreted. diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 4266ffd..039ec8e 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -221,7 +221,7 @@ const struct bpf_func_proto bpf_get_current_comm_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_CONST_SIZE32, }; #if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3bcbd9b..83b3f83 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1174,12 +1174,47 @@ static int mark_reg_read(struct bpf_verifier_env *env, return 0; } +static bool helper_call_arg64(struct bpf_verifier_env *env, int func_id, + u32 regno) +{ + /* get_func_proto must succeed, other it should have been rejected + * early inside check_helper_call. + */ + const struct bpf_func_proto *fn = + env->ops->get_func_proto(func_id, env->prog); + enum bpf_arg_type arg_type; + + switch (regno) { + case BPF_REG_1: + arg_type = fn->arg1_type; + break; + case BPF_REG_2: + arg_type = fn->arg2_type; + break; + case BPF_REG_3: + arg_type = fn->arg3_type; + break; + case BPF_REG_4: + arg_type = fn->arg4_type; + break; + case BPF_REG_5: + arg_type = fn->arg5_type; + break; + default: + arg_type = ARG_DONTCARE; + } + + return arg_type != ARG_CONST_SIZE32 && + arg_type != ARG_CONST_SIZE32_OR_ZERO && + arg_type != ARG_ANYTHING32; +} + /* This function is supposed to be used by the following 32-bit optimization * code only. It returns TRUE if the source or destination register operates * on 64-bit, otherwise return FALSE. */ -static bool is_reg64(struct bpf_insn *insn, u32 regno, - struct bpf_reg_state *reg, enum reg_arg_type t) +static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, + u32 regno, struct bpf_reg_state *reg, enum reg_arg_type t) { u8 code, class, op; @@ -1201,9 +1236,12 @@ static bool is_reg64(struct bpf_insn *insn, u32 regno, if (insn->src_reg == BPF_PSEUDO_CALL) return false; /* Helper call will reach here because of arg type - * check. Conservatively marking all args as 64-bit. + * check. */ - return true; + if (t == SRC_OP) + return helper_call_arg64(env, insn->imm, regno); + + return false; } } @@ -1283,7 +1321,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, } reg = ®s[regno]; - rw64 = is_reg64(insn, regno, reg, t); + rw64 = is_reg64(env, insn, regno, reg, t); if (t == SRC_OP) { /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { @@ -2576,7 +2614,9 @@ static bool arg_type_is_mem_ptr(enum bpf_arg_type type) static bool arg_type_is_mem_size(enum bpf_arg_type type) { return type == ARG_CONST_SIZE || - type == ARG_CONST_SIZE_OR_ZERO; + type == ARG_CONST_SIZE_OR_ZERO || + type == ARG_CONST_SIZE32 || + type == ARG_CONST_SIZE32_OR_ZERO; } static bool arg_type_is_int_ptr(enum bpf_arg_type type) @@ -2610,7 +2650,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, if (err) return err; - if (arg_type == ARG_ANYTHING) { + if (arg_type == ARG_ANYTHING || arg_type == ARG_ANYTHING32) { if (is_pointer_value(env, regno)) { verbose(env, "R%d leaks addr into helper function\n", regno); @@ -2633,7 +2673,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, type != expected_type) goto err_type; } else if (arg_type == ARG_CONST_SIZE || - arg_type == ARG_CONST_SIZE_OR_ZERO) { + arg_type == ARG_CONST_SIZE_OR_ZERO || + arg_type == ARG_CONST_SIZE32 || + arg_type == ARG_CONST_SIZE32_OR_ZERO) { expected_type = SCALAR_VALUE; if (type != expected_type) goto err_type; @@ -2733,7 +2775,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, meta->map_ptr->value_size, false, meta); } else if (arg_type_is_mem_size(arg_type)) { - bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO); + bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO || + arg_type == ARG_CONST_SIZE32_OR_ZERO); /* remember the mem_size which may be used later * to refine return values. diff --git a/net/core/filter.c b/net/core/filter.c index 95a27fd..d3c7200 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -1694,9 +1694,9 @@ static const struct bpf_func_proto bpf_skb_store_bytes_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_PTR_TO_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_CONST_SIZE32, .arg5_type = ARG_ANYTHING, }; @@ -1725,9 +1725,9 @@ static const struct bpf_func_proto bpf_skb_load_bytes_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_CONST_SIZE32, }; BPF_CALL_5(bpf_skb_load_bytes_relative, const struct sk_buff *, skb, @@ -1876,7 +1876,7 @@ static const struct bpf_func_proto bpf_l3_csum_replace_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, @@ -1929,7 +1929,7 @@ static const struct bpf_func_proto bpf_l4_csum_replace_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, @@ -1968,9 +1968,9 @@ static const struct bpf_func_proto bpf_csum_diff_proto = { .pkt_access = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM_OR_NULL, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_CONST_SIZE32_OR_ZERO, .arg3_type = ARG_PTR_TO_MEM_OR_NULL, - .arg4_type = ARG_CONST_SIZE_OR_ZERO, + .arg4_type = ARG_CONST_SIZE32_OR_ZERO, .arg5_type = ARG_ANYTHING, }; @@ -2151,7 +2151,7 @@ static const struct bpf_func_proto bpf_redirect_proto = { .func = bpf_redirect, .gpl_only = false, .ret_type = RET_INTEGER, - .arg1_type = ARG_ANYTHING, + .arg1_type = ARG_ANYTHING32, .arg2_type = ARG_ANYTHING, }; @@ -2929,7 +2929,7 @@ static const struct bpf_func_proto bpf_skb_change_proto_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, }; @@ -2949,7 +2949,7 @@ static const struct bpf_func_proto bpf_skb_change_type_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, }; static u32 bpf_skb_net_base_len(const struct sk_buff *skb) @@ -3241,7 +3241,7 @@ static const struct bpf_func_proto bpf_skb_change_tail_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, }; @@ -3837,7 +3837,7 @@ static const struct bpf_func_proto bpf_skb_get_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_CONST_SIZE32, .arg4_type = ARG_ANYTHING, }; @@ -3946,7 +3946,7 @@ static const struct bpf_func_proto bpf_skb_set_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_CONST_SIZE32, .arg4_type = ARG_ANYTHING, }; From patchwork Fri Apr 12 21:59:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084943 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="acmJANmk"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJc06gLz9sD4 for ; Sat, 13 Apr 2019 08:00:16 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727209AbfDLWAO (ORCPT ); Fri, 12 Apr 2019 18:00:14 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:39018 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727201AbfDLWAO (ORCPT ); Fri, 12 Apr 2019 18:00:14 -0400 Received: by mail-wm1-f65.google.com with SMTP id n25so12639679wmk.4 for ; Fri, 12 Apr 2019 15:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jdVahutpxLcDXWoGIWnyKYQilj6YT55NUFsmFdjSUl4=; b=acmJANmkCvNsbHbiqjPC2vPsN3DO7NzwarjYH7Bv5U9zyYBmCszJUMFx2oZ5h+vjnv WJfi18JSMU1VC1O1BJ7/C8oDxzbnc/xY63zEYU78/TINk6Mb0lAiXJwKcDbk4GnCt42Z s19kuIinIP4j/QwTNO9yBFmAd0QaU2iSRTWxn3URq6OUTnAWY7EKsMfjoZ8gFkNQzS0T 94E1uAXu/N949w0us/j5RYzcwZQLeRYammy8qAdHmg8SNF0+YQiAQin/9JCbsn1a1Bfr 5GWDZ+XQjLC9vKSsWxOKRPMF2ycf4uBzvlgYLZQEf3coGBlihI7iOxIY/GmTxpCuAVpr 3vYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jdVahutpxLcDXWoGIWnyKYQilj6YT55NUFsmFdjSUl4=; b=EZTwAa4tzAlGKIX4Y+QPrvXWKS6QnOhF+iRLmVtID0hAP2Xc4cPVfc6B31zr/sySxF CJ+YTqW5lB5UKjHuIxj8TkG9UUkqP8qtwskgxjN5HKKCzt3UO/2uhG6s+qx2bV2fBkhu 9+iK1IZEbwSbGQXa+4qtWwFnQjeFdrSA/ENie9WepSK8W57k9ex8kp8hrmyX7iEAzHyj 8gmfWX0KxXEcGL9sh5H5C9dpJOR/bOna+Ki6mppLbVpN7Lj2wKya2e29HB4zegDIHcuG DVwfx1fNagMzm2JYuwBJlX7vAhqy6zCtf6hRoYfMMf8N+U2o34qYvGlA/O8RGMRHCPWG RAog== X-Gm-Message-State: APjAAAU2XhpF9uyzFe7dIm5Mt2E5+MzfE9NGdx+uX8WGeCWr/uawNSgv zZOvsbjjPcx6HxylQekWhBZetA== X-Google-Smtp-Source: APXvYqyiUDmIZketKuOnViLkwV8HJ67sxnHsjfs/CwnVkti7c3rS9iPTXSdWEHNinBWa+no4gIuggw== X-Received: by 2002:a1c:6c09:: with SMTP id h9mr12544254wmc.130.1555106411213; Fri, 12 Apr 2019 15:00:11 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:10 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 08/19] bpf: insert explicit zero extension insn when hardware doesn't do it implicitly Date: Fri, 12 Apr 2019 22:59:41 +0100 Message-Id: <1555106392-20117-9-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org After previous patches, verifier has marked those instructions that really need zero extension on dst_reg. It is then for all back-ends to decide how to use such information to eliminate unnecessary zero extension code-gen during JIT compilation. One approach is: 1. Verifier insert explicit zero extension for those instructions that need zero extension. 2. All JIT back-ends do NOT generate zero extension for sub-register write any more. The good thing for this approach is no major change on JIT back-end interface, all back-ends could get this optimization. However, only those back-ends that do not have hardware zero extension want this optimization. For back-ends like x86_64 and AArch64, there is hardware support, so zext insertion should be disabled. This patch introduces new target hook "bpf_jit_hardware_zext" which is default true, meaning the underlying hardware will do zero extension implicitly, therefore zext insertion by verifier will be disabled. Once a back-end overrides this hook to false, then verifier will insert zext sequence to clear high 32-bit of definitions when necessary. Offload targets do not use this native target hook, instead, they could get the optimization results using bpf_prog_offload_ops.finalize. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf.h | 1 + include/linux/filter.h | 1 + kernel/bpf/core.c | 8 +++++ kernel/bpf/verifier.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 96 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 884b8e1..bdab6e7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -368,6 +368,7 @@ struct bpf_prog_aux { u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */ + bool no_verifier_zext; /* No zero extension insertion by verifier. */ bool offload_requested; struct bpf_prog **func; void *jit_data; /* JIT specific data. arch dependent */ diff --git a/include/linux/filter.h b/include/linux/filter.h index fb0edad..8750657 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -821,6 +821,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); +bool bpf_jit_hardware_zext(void); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(void) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2792eda..1c54274 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2091,6 +2091,14 @@ bool __weak bpf_helper_changes_pkt_data(void *func) return false; } +/* Return TRUE is the target hardware of JIT will do zero extension to high bits + * when writing to low 32-bit of one register. Otherwise, return FALSE. + */ +bool __weak bpf_jit_hardware_zext(void) +{ + return true; +} + /* To execute LD_ABS/LD_IND instructions __bpf_prog_run() may call * skb_copy_bits(), so provide a weak definition of it for NET-less config. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 83b3f83..016f81d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7551,6 +7551,80 @@ static int opt_remove_nops(struct bpf_verifier_env *env) return 0; } +static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) +{ + struct bpf_insn_aux_data orig_aux, *aux = env->insn_aux_data; + struct bpf_insn *insns = env->prog->insnsi; + int i, delta = 0, len = env->prog->len; + struct bpf_insn zext_patch[3]; + struct bpf_prog *new_prog; + + zext_patch[1] = BPF_ALU64_IMM(BPF_LSH, 0, 32); + zext_patch[2] = BPF_ALU64_IMM(BPF_RSH, 0, 32); + for (i = 0; i < len; i++) { + int adj_idx = i + delta; + struct bpf_insn insn; + + if (!aux[adj_idx].zext_dst) + continue; + + insn = insns[adj_idx]; + /* "adjust_insn_aux_data" only retains the original insn aux + * data if insn at patched offset is at the end of the patch + * buffer. That is to say, given the following insn sequence: + * + * insn 1 + * insn 2 + * insn 3 + * + * if the patch offset is at insn 2, then the patch buffer must + * be the following that original insn aux data can be retained. + * + * {lshift, rshift, insn2} + * + * However, zero extension needs to be inserted after insn2, so + * insn patch buffer needs to be the following: + * + * {insn2, lshift, rshift} + * + * which would cause insn aux data of insn2 lost and that data + * is critical for ctx field load instruction transformed + * correctly later inside "convert_ctx_accesses". + * + * The simplest way to fix this to build the following patch + * buffer: + * + * {lshift, rshift, insn-next-to-insn2} + * + * Given insn2 defines a value, it can't be a JMP, hence there + * must be a next insn for it otherwise CFG check should have + * rejected this program. However, insn-next-to-insn2 could + * be a JMP and verifier insn patch infrastructure doesn't + * support adjust offset for JMP inside patch buffer. We would + * end up with a few insn check and offset adj code outside of + * the generic insn patch helpers if we go with this approach. + * + * Therefore, we still use {insn2, lshift, rshift} as the patch + * buffer, we copy and restore insn aux data for insn2 + * explicitly. The change looks simpler and smaller. + */ + zext_patch[0] = insns[adj_idx]; + zext_patch[1].dst_reg = insn.dst_reg; + zext_patch[2].dst_reg = insn.dst_reg; + memcpy(&orig_aux, &aux[adj_idx], sizeof(orig_aux)); + new_prog = bpf_patch_insn_data(env, adj_idx, zext_patch, 3); + if (!new_prog) + return -ENOMEM; + env->prog = new_prog; + insns = new_prog->insnsi; + aux = env->insn_aux_data; + memcpy(&aux[adj_idx], &orig_aux, sizeof(orig_aux)); + delta += 2; + } + + return 0; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -8382,7 +8456,18 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) ret = check_max_stack_depth(env); - /* instruction rewrites happen after this point */ + /* Instruction rewrites happen after this point. + * For offload target, finalize hook has all aux insn info, do any + * customized work there. + */ + if (ret == 0 && !bpf_jit_hardware_zext() && + !bpf_prog_is_dev_bound(env->prog->aux)) { + ret = opt_subreg_zext_lo32(env); + env->prog->aux->no_verifier_zext = !!ret; + } else { + env->prog->aux->no_verifier_zext = true; + } + if (is_priv) { if (ret == 0) opt_hard_wire_dead_code_branches(env); From patchwork Fri Apr 12 21:59:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084968 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="SDh96Nim"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsK96cb6z9s71 for ; Sat, 13 Apr 2019 08:00:45 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727339AbfDLWAm (ORCPT ); Fri, 12 Apr 2019 18:00:42 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:35368 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727208AbfDLWAO (ORCPT ); Fri, 12 Apr 2019 18:00:14 -0400 Received: by mail-wr1-f66.google.com with SMTP id w1so13732219wrp.2 for ; Fri, 12 Apr 2019 15:00:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=J0YL3VY81ZYkLm7FFiGr50XaGEO0Un8tKpiZShOimgc=; b=SDh96NimQGVnmCD3G9uU2gLWVqjy9PUzYzOKCNA6GsUdgbpKHh1HXSxq2//xnxe2kq bae3Mb6V5JVSgLVbS81jJElc8tXGib5xL3QCGfgDT5OsJhK7pV0uD10aS67CQqGOEmwb Wgf94ETAYN5ZKh1rAFfUGqvdN6iaX49o063lfDMkp1kqbrV1IstcaEaLGMYfkg9EShp4 vInfSrheI4VLH6mgzkECRduXU9JPOLbqeKLd4t+CVXBJoEscKds60Lgmxy9MT4yQKWzC qVwKITwc9TXnce39DbY893xlsxGHwpi3e9K3L7QBr8Bm73enR+itsq31ujGt9KQ9HuDg H2cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=J0YL3VY81ZYkLm7FFiGr50XaGEO0Un8tKpiZShOimgc=; b=A9eTjeyfOeAz0I9HSIr3UTyvE9vuDkOOtx2hxn0KCE+rbj9llRk+3NWSbB27xVbmrN jGVpk/eD7NEn0AsfmmIDFKAocEgUER9bOlgxVSs5uedNf/TBj7KiqeUHrULVwaQR+eWU R8Aq4lEy3a2D8vA92qo0XVOnK/3AlfpSse2voG9aATOPxgI3Mthp+6cD6UQ4kR3eYdUZ Vr+DdH7efgdHCVFNSXlC1AdAwSadTwENWd0PJc8fvNHZHObiHYRelP3H3w+XrZ3appYs YIAzW1W5uJbF3akRSG2y0Xn4MltDyyEC/kaUoaaoD9dsyh2BXJnzy2KNRg32Qdj+QEPG +ZqA== X-Gm-Message-State: APjAAAXUdJNMYgFJS6A9uRIXMVeYRP1dGMJ6GFdnipVf7HpLrI8gsVct ioBUPuNTsB+uK65yzxjc74n+jA== X-Google-Smtp-Source: APXvYqyXd9E5Vkbz6d0Fzbn/JzQzT/dPfV/l1mQlr1XZLg08V7PQp2BfBlAq1/bw/DW+iC8ZYMyz9w== X-Received: by 2002:adf:fec3:: with SMTP id q3mr38000727wrs.173.1555106412288; Fri, 12 Apr 2019 15:00:12 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:11 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 09/19] bpf: introduce new bpf prog load flags "BPF_F_TEST_RND_HI32" Date: Fri, 12 Apr 2019 22:59:42 +0100 Message-Id: <1555106392-20117-10-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org x86_64 and AArch64 perhaps are two arches that running bpf testsuite frequently, however the zero extension insertion pass is not enabled for them because of their hardware support. It is critical to guarantee the pass correction as it is supposed to be enabled at default for a couple of other arches, for example PowerPC, SPARC, arm, NFP etc. Therefore, it would be very useful if there is a way to test this pass on for example x86_64. The test methodology employed by this set is "poisoning" useless bits. High 32-bit of a definition is randomized if it is identified as not used by any later instructions. Such randomization is only enabled under testing mode which is gated by the new bpf prog load flags "BPF_F_TEST_RND_HI32". Suggested-by: Alexei Starovoitov Signed-off-by: Jiong Wang --- include/uapi/linux/bpf.h | 18 ++++++++++++++++++ kernel/bpf/syscall.c | 4 +++- tools/include/uapi/linux/bpf.h | 18 ++++++++++++++++++ 3 files changed, 39 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c26be24..89c85a3 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -258,6 +258,24 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) +/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose. + * Verifier does sub-register def/use analysis and identifies instructions whose + * def only matters for low 32-bit, high 32-bit is never referenced later + * through implicit zero extension. Therefore verifier notifies JIT back-ends + * that it is safe to ignore clearing high 32-bit for these instructions. This + * saves some back-ends a lot of code-gen. However such optimization is not + * necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends + * hence hasn't used verifier's analysis result. But, we really want to have a + * way to be able to verify the correctness of the described optimization on + * x86_64 on which testsuites are frequently exercised. + * + * So, this flag is introduced. Once it is set, verifier will randomize high + * 32-bit for those instructions who has been identified as safe to ignore them. + * Then, if verifier is not doing correct analysis, such randomization will + * regress tests to expose bugs. + */ +#define BPF_F_TEST_RND_HI32 (1U << 2) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * two extensions: * diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 92c9b8a..abe2804 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1600,7 +1600,9 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) if (CHECK_ATTR(BPF_PROG_LOAD)) return -EINVAL; - if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT | BPF_F_ANY_ALIGNMENT)) + if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT | + BPF_F_ANY_ALIGNMENT | + BPF_F_TEST_RND_HI32)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c26be24..89c85a3 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -258,6 +258,24 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) +/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose. + * Verifier does sub-register def/use analysis and identifies instructions whose + * def only matters for low 32-bit, high 32-bit is never referenced later + * through implicit zero extension. Therefore verifier notifies JIT back-ends + * that it is safe to ignore clearing high 32-bit for these instructions. This + * saves some back-ends a lot of code-gen. However such optimization is not + * necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends + * hence hasn't used verifier's analysis result. But, we really want to have a + * way to be able to verify the correctness of the described optimization on + * x86_64 on which testsuites are frequently exercised. + * + * So, this flag is introduced. Once it is set, verifier will randomize high + * 32-bit for those instructions who has been identified as safe to ignore them. + * Then, if verifier is not doing correct analysis, such randomization will + * regress tests to expose bugs. + */ +#define BPF_F_TEST_RND_HI32 (1U << 2) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * two extensions: * From patchwork Fri Apr 12 21:59:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084946 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="HnLnH6xD"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJf1NQmz9sD4 for ; Sat, 13 Apr 2019 08:00:18 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727224AbfDLWAQ (ORCPT ); Fri, 12 Apr 2019 18:00:16 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:39021 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727201AbfDLWAQ (ORCPT ); Fri, 12 Apr 2019 18:00:16 -0400 Received: by mail-wm1-f65.google.com with SMTP id n25so12639764wmk.4 for ; Fri, 12 Apr 2019 15:00:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PmiIckmFI3rbvt6JZnWc1hJk0xLs+56oTMLQArAslWc=; b=HnLnH6xDobFCl99EyWHuDkcFzUGX7zKHTKh4Tk4CxibIt/MmgUudVgqpv4/6aw6yBE DiA1igBL14S3MBFs5IlHC2GCb1oTEK764zyQyMUoBfm/47QJrtvsZtlJQVXqnHpk7zpJ 0E4BKUC2mdp9AGzf9dQ13eRINCJX+WO1qnCC1uryrRuWIeaFZ6tp5VFIzUud2NvHRyct iONRupys9pNd1/Z4AGYBdHFMd94tC68TTEgl4GiDxAAOIqNIGyYvSiyTSQVQNshLjA4z BoGO31yW6woMISfPWFydjmHT6XChlBhgKHKfzZ1+T30HS6pBizI9fhCaqgaEAlpmrfjv vIcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PmiIckmFI3rbvt6JZnWc1hJk0xLs+56oTMLQArAslWc=; b=BPvlRCJcuQMWJWu1PpZR4HuZvLzEWtSaPLCHib6RdTjXk09p0k6Ss7KXq1FJIU120r 7H6Ohlm/yH+NxQRsx1tLxo0HrmrEU5EkAe7Re9mt2io5px7RFIVqKQvKGNKg20rR6RrP HeT3SjiULZLNowVl/NsXq0Kajx27iLFRLi5La0ulFjIaqFnImgpheeXmMMkP+K13ONjV G2fy3fXrl2icNiRa5PD10sEEopF6J6aPoepvTjrGhM5DTgDNnrNcqXr5uPw6GLs8txuL Ep8HJRdRqQQe8wHgKv8MHil0M7hilHHvN47v7+8tZKW8Jd4MHMGRZ9QC5zBwdF7lgLUv RzFg== X-Gm-Message-State: APjAAAW1FV3i659VPjkvlnkv8IZ3f3UwAosdwEwRP/uELoNf/uc1Qf1o lEoEz9eQ1/rxkjEUEkX1bxx7Ww== X-Google-Smtp-Source: APXvYqy0rhCbTMNxAdLHYIQv/+ZRxvwmAnQHD5eyBns6YdmlPtHEYQ223XNnJ/Fs+vpLRmlyP4R37A== X-Received: by 2002:a1c:1a46:: with SMTP id a67mr13076401wma.21.1555106413255; Fri, 12 Apr 2019 15:00:13 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:12 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 10/19] bpf: randomize high 32-bit when BPF_F_TEST_RND_HI32 is set Date: Fri, 12 Apr 2019 22:59:43 +0100 Message-Id: <1555106392-20117-11-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org This patch randomizes high 32-bit of a definition when BPF_F_TEST_RND_HI32 is set. It does this once the flag set no matter there is hardware zero extension support or not. Because this is a test feature and we want to deliver the most stressful test. Suggested-by: Alexei Starovoitov Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 85 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 016f81d..eb00232 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7551,24 +7551,70 @@ static int opt_remove_nops(struct bpf_verifier_env *env) return 0; } -static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) +static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, + const union bpf_attr *attr) { struct bpf_insn_aux_data orig_aux, *aux = env->insn_aux_data; + struct bpf_insn *patch, zext_patch[3], rnd_hi32_patch[4]; + int i, patch_len, delta = 0, len = env->prog->len; struct bpf_insn *insns = env->prog->insnsi; - int i, delta = 0, len = env->prog->len; - struct bpf_insn zext_patch[3]; struct bpf_prog *new_prog; + bool rnd_hi32; + + rnd_hi32 = attr->prog_flags & BPF_F_TEST_RND_HI32; zext_patch[1] = BPF_ALU64_IMM(BPF_LSH, 0, 32); zext_patch[2] = BPF_ALU64_IMM(BPF_RSH, 0, 32); + rnd_hi32_patch[1] = BPF_ALU64_IMM(BPF_MOV, BPF_REG_AX, 0); + rnd_hi32_patch[2] = BPF_ALU64_IMM(BPF_LSH, BPF_REG_AX, 32); + rnd_hi32_patch[3] = BPF_ALU64_REG(BPF_OR, 0, BPF_REG_AX); for (i = 0; i < len; i++) { int adj_idx = i + delta; struct bpf_insn insn; - if (!aux[adj_idx].zext_dst) + insn = insns[adj_idx]; + if (!aux[adj_idx].zext_dst) { + u8 code, class; + u32 imm_rnd; + + if (!rnd_hi32) + continue; + + code = insn.code; + class = BPF_CLASS(code); + /* Insns doesn't define any value. */ + if (class == BPF_JMP || class == BPF_JMP32 || + class == BPF_STX || class == BPF_ST) + continue; + + /* NOTE: arg "reg" is only used for BPF_STX, as it has + * been ruled out in above check, it is safe to + * pass NULL here. + */ + if (is_reg64(env, &insn, insn.dst_reg, NULL, DST_OP)) { + if (class == BPF_LD && + BPF_MODE(code) == BPF_IMM) + i++; + continue; + } + + /* ctx load could be transformed into wider load. */ + if (class == BPF_LDX && + aux[adj_idx].ptr_type == PTR_TO_CTX) + continue; + + imm_rnd = get_random_int(); + rnd_hi32_patch[0] = insns[adj_idx]; + rnd_hi32_patch[1].imm = imm_rnd; + rnd_hi32_patch[3].dst_reg = insn.dst_reg; + patch = rnd_hi32_patch; + patch_len = 4; + goto apply_patch_buffer; + } + + if (bpf_jit_hardware_zext()) continue; - insn = insns[adj_idx]; /* "adjust_insn_aux_data" only retains the original insn aux * data if insn at patched offset is at the end of the patch * buffer. That is to say, given the following insn sequence: @@ -7611,15 +7657,18 @@ static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) zext_patch[0] = insns[adj_idx]; zext_patch[1].dst_reg = insn.dst_reg; zext_patch[2].dst_reg = insn.dst_reg; + patch = zext_patch; + patch_len = 3; +apply_patch_buffer: memcpy(&orig_aux, &aux[adj_idx], sizeof(orig_aux)); - new_prog = bpf_patch_insn_data(env, adj_idx, zext_patch, 3); + new_prog = bpf_patch_insn_data(env, adj_idx, patch, patch_len); if (!new_prog) return -ENOMEM; env->prog = new_prog; insns = new_prog->insnsi; aux = env->insn_aux_data; memcpy(&aux[adj_idx], &orig_aux, sizeof(orig_aux)); - delta += 2; + delta += patch_len - 1; } return 0; @@ -8456,16 +8505,18 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) ret = check_max_stack_depth(env); - /* Instruction rewrites happen after this point. - * For offload target, finalize hook has all aux insn info, do any - * customized work there. - */ - if (ret == 0 && !bpf_jit_hardware_zext() && - !bpf_prog_is_dev_bound(env->prog->aux)) { - ret = opt_subreg_zext_lo32(env); - env->prog->aux->no_verifier_zext = !!ret; - } else { - env->prog->aux->no_verifier_zext = true; + /* Instruction rewrites happen after this point. */ + if (ret == 0) { + if (bpf_prog_is_dev_bound(env->prog->aux)) { + /* For offload target, finalize hook has all aux insn + * info, copy the analysis result at there. + */ + env->prog->aux->no_verifier_zext = true; + } else { + ret = opt_subreg_zext_lo32_rnd_hi32(env, attr); + env->prog->aux->no_verifier_zext = + bpf_jit_hardware_zext() ? true : !!ret; + } } if (is_priv) { From patchwork Fri Apr 12 21:59:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084961 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="As6J63d8"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJz18pkz9s8m for ; Sat, 13 Apr 2019 08:00:35 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727264AbfDLWAT (ORCPT ); Fri, 12 Apr 2019 18:00:19 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:43811 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727250AbfDLWAQ (ORCPT ); Fri, 12 Apr 2019 18:00:16 -0400 Received: by mail-wr1-f67.google.com with SMTP id k17so9057587wrx.10 for ; Fri, 12 Apr 2019 15:00:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=33/0vLDNV+/fkdAJJtAOFzRJSQIBJATreCk8IP7/Y7g=; b=As6J63d8ugwdyObljS0PPYV0EnjNNkhhD8QpjYbzD50y0RivvBVWKZCwnty/QzQn8Q k3MXdLXXH4ZHC4tx0Frbl94xQoTEZeNeohp/9z5A3kPkrq28qYGDZc3asPClCkbTRF49 LekKAOgwYvMGOJq7GUG4HiONbvFlzpYIEZz9kvSXb/TCX6qZL6I/pQ7M4P5TGdo14Vxo c4aB7/mSzJZ1wZZgpSvPq/c1wdFjqGC6UlokE5l3ZNmG/0Bx/4PQWlQ+4YlBaVBZbK7Z wSqfTRTVc47woqYO/FgDM1FfwB0TnVFhYrpfziBBnOTMesWQzwvzE62ezbMLKwXHatu7 XLjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=33/0vLDNV+/fkdAJJtAOFzRJSQIBJATreCk8IP7/Y7g=; b=Bj4jCKWvK+KW0XkBfrkm+E1gDNPWp+DZjEztfj6c+StX//cuHYHaCcVxFeaS+izgA9 HCJm/csX+3Vbyua9BtWfSzS4DofdAenYQLP2nKe3wJKIRusKC4AmKSlX02Epr6CClMGr roT4S1FgSxZKBq2tkallrd1XgOXlfq4EWRGGheSf8XAxLNKVIFOCWWwgD21x026yy0Ms 7ujF/EvO8ChTnrqfuxK3bCvZxWmawSUtOP8loArCs6F4el4v2NnWqV5YYq/8IzwiMOdT YHt+4ELJiXLAONMHKXzeDLQcy+Jqdt9+tRKrvqUONhXTZnDUi+H4bXGuseA5c54Oqzii mf+A== X-Gm-Message-State: APjAAAVnU09Cc7snTfHfdW1onDBcUzYHQHoOid8e46MC/F760zdtYYTf O4QVdC94gR2sJmf/9Wv61ZIV3g== X-Google-Smtp-Source: APXvYqzd1Efb/GBemCc+bWtHozRU0zvlJT+Mw0RxvPX+qFpZu8DXXsFZaQUd4R5UiYukyqidtygPfg== X-Received: by 2002:a5d:4750:: with SMTP id o16mr34176244wrs.206.1555106414206; Fri, 12 Apr 2019 15:00:14 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:13 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 11/19] libbpf: add "prog_flags" to bpf_program/bpf_prog_load_attr/bpf_load_program_attr Date: Fri, 12 Apr 2019 22:59:44 +0100 Message-Id: <1555106392-20117-12-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org libbpf doesn't allow passing "prog_flags" during bpf program load in a couple of load related APIs, "bpf_load_program_xattr", "load_program" and "bpf_prog_load_xattr". It makes sense to allow passing "prog_flags" which is useful for customizing program loading. Signed-off-by: Jiong Wang Reviewed-by: Jakub Kicinski --- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/bpf.h | 1 + tools/lib/bpf/libbpf.c | 3 +++ tools/lib/bpf/libbpf.h | 1 + 4 files changed, 6 insertions(+) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 955191c..f79ec49 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -254,6 +254,7 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, if (load_attr->name) memcpy(attr.prog_name, load_attr->name, min(strlen(load_attr->name), BPF_OBJ_NAME_LEN - 1)); + attr.prog_flags = load_attr->prog_flags; fd = sys_bpf_prog_load(&attr, sizeof(attr)); if (fd >= 0) diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index bc30783..a983442 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -86,6 +86,7 @@ struct bpf_load_program_attr { const void *line_info; __u32 line_info_cnt; __u32 log_level; + __u32 prog_flags; }; /* Flags to direct loading requirements */ diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index e5b77ad..e0affd0 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -182,6 +182,7 @@ struct bpf_program { void *line_info; __u32 line_info_rec_size; __u32 line_info_cnt; + __u32 prog_flags; }; enum libbpf_map_type { @@ -1876,6 +1877,7 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, load_attr.line_info_rec_size = prog->line_info_rec_size; load_attr.line_info_cnt = prog->line_info_cnt; load_attr.log_level = prog->log_level; + load_attr.prog_flags = prog->prog_flags; if (!load_attr.insns || !load_attr.insns_cnt) return -EINVAL; @@ -3320,6 +3322,7 @@ int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, expected_attach_type); prog->log_level = attr->log_level; + prog->prog_flags = attr->prog_flags; if (!first_prog) first_prog = prog; } diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index c5ff005..5abc237 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -320,6 +320,7 @@ struct bpf_prog_load_attr { enum bpf_attach_type expected_attach_type; int ifindex; int log_level; + int prog_flags; }; LIBBPF_API int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, From patchwork Fri Apr 12 21:59:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084948 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="hpyzZa6g"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJg1BH4z9sNC for ; Sat, 13 Apr 2019 08:00:19 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727276AbfDLWAS (ORCPT ); Fri, 12 Apr 2019 18:00:18 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:42363 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727260AbfDLWAR (ORCPT ); Fri, 12 Apr 2019 18:00:17 -0400 Received: by mail-wr1-f68.google.com with SMTP id g3so13700145wrx.9 for ; Fri, 12 Apr 2019 15:00:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WoOM3rYmPYJxVZhgrctnlEaRiC1Bq/772bv5PR4BN8E=; b=hpyzZa6gdCiIZ6B/85Gc/ZqH40ASPU8/NH9FehNlbGbuZo1g9Bbm2iVpHOX3Gv8ohP yp0AsU0kATsGAO+9tcT2+czBs9gr8vk72z5mHHI/XJp34g0Cj7nvPPC9bSIw6N6QHOVf J2ppEvnwwYglfacDmRBtfcybs27qGk/liLBkAkrg0iflEHy2sNJnC59hL7BQYFTug4n2 puq57/ybei0bx2Y25SFYV4whpyQ+y1NWx+0BMRKHXGfyB6dGd8g20bzM48iB8yDn8FcN 7F5vRa6Fs2XwJDb23vFsAjVg2dfaTxnTttdfwfR/yKeDMQXCqIClryEXO/1YggqpccNi TTBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WoOM3rYmPYJxVZhgrctnlEaRiC1Bq/772bv5PR4BN8E=; b=g9eoomNPsDzQluy4tIKAENw65CTlQ9914D8+hrTmJCGVDu5kK2H+Wa8446O7qYil/5 /A8Mb/vWkBuTyQAIFsVl3SRc68u+bLVz15kmAxX5uHBwfgNaqeUXP/HMZkv1hb5M/+8J ryHk0XLrFqdHKWuds+Tjxn6VPvupTmJ9/xLzyR4tNB8+kF7p8CDYFWXTISvmAs7o6RTC UAFcYNhJGO0UzRpkO15YxezTyykjzytRDpDzDl8HfLSWI6MPgkR0HOe1soj/odPCSncH yozdNsSbsDc/iAVG5MP+z7+qAZJJXTP2pUftczdovXqBEYdnY1AWsjQnOVJSPkIxWtMi bbvg== X-Gm-Message-State: APjAAAVPmyXbn42LhZ+QI8n6v59AlYpI0eEUmqY0Gbe4ba9CHr2+BtOK 51Xaa3LPCjoZHJkKzK4q4EPWyw== X-Google-Smtp-Source: APXvYqyP3Ah/v+Xpa+p3+6qRKIlzcW2CVVigPp/1noSzm1BSvreEJEejJGsqwMvM34exXapedQ1Y+g== X-Received: by 2002:adf:f68d:: with SMTP id v13mr36674888wrp.6.1555106415349; Fri, 12 Apr 2019 15:00:15 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:14 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 12/19] selftests: enable hi32 randomization for all tests Date: Fri, 12 Apr 2019 22:59:45 +0100 Message-Id: <1555106392-20117-13-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org The previous libbpf patch allows user to specify "prog_flags" to bpf program load APIs. To enable high 32-bit randomization for a test, we need to set BPF_F_TEST_RND_HI32 in "prog_flags". To enable such randomization for all tests, we need to make sure all places are passing BPF_F_TEST_RND_HI32. Changing them one by one is not convenient, also, it would be better if a test could be switched to "normal" running mode without code change. Given the program load APIs used across bpf selftests are mostly: bpf_prog_load: load from file bpf_load_program: load from raw insns A test_stub.c is implemented for bpf seltests, it offers two functions for testing purpose: bpf_prog_test_load bpf_test_load_program The are the same as "bpf_prog_load" and "bpf_load_program", except they also set BPF_F_TEST_RND_HI32. Given *_xattr functions are the APIs to customize any "prog_flags", it makes little sense to put these two functions into libbpf. Then, the following CFLAGS are passed to compilations for host programs: -Dbpf_prog_load=bpf_prog_test_load -Dbpf_load_program=bpf_test_load_program They migrate the used load APIs to the test version, hence enable high 32-bit randomization for these tests without changing source code. "test_verifier" is using bpf_verify_program which supports passing "prog_flags" already, so we are fine. But, two unit tests needs to be adjusted due to there will be 16-bit jump distance overflow: - "ld_abs: vlan + abs, test1" The function "bpf_fill_ld_abs_vlan_push_pop" inside test_verifier.c needs to use ALU64 to avoid insertion of hi32 randomization sequence that would overflow the sequence. - bpf_fill_jump_around_ld_abs needs to consider hi32 randomization to the load dst of ld_abs. Besides all these, there are several testcases are using "bpf_prog_load_attr" directly, their call sites are updated to pass BPF_F_TEST_RND_HI32. Signed-off-by: Jiong Wang --- tools/testing/selftests/bpf/Makefile | 10 +++--- .../selftests/bpf/prog_tests/bpf_verif_scale.c | 1 + tools/testing/selftests/bpf/test_sock_addr.c | 1 + tools/testing/selftests/bpf/test_sock_fields.c | 1 + tools/testing/selftests/bpf/test_socket_cookie.c | 1 + tools/testing/selftests/bpf/test_stub.c | 40 ++++++++++++++++++++++ tools/testing/selftests/bpf/test_verifier.c | 6 ++-- 7 files changed, 53 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/test_stub.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index f9d83ba..f2accf6 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -15,7 +15,9 @@ LLC ?= llc LLVM_OBJCOPY ?= llvm-objcopy LLVM_READELF ?= llvm-readelf BTF_PAHOLE ?= pahole -CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(BPFDIR) -I$(GENDIR) $(GENFLAGS) -I../../../include +CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(BPFDIR) -I$(GENDIR) $(GENFLAGS) -I../../../include \ + -Dbpf_prog_load=bpf_prog_test_load \ + -Dbpf_load_program=bpf_test_load_program LDLIBS += -lcap -lelf -lrt -lpthread # Order correspond to 'make run_tests' order @@ -76,9 +78,9 @@ $(OUTPUT)/urandom_read: $(OUTPUT)/%: %.c BPFOBJ := $(OUTPUT)/libbpf.a -$(TEST_GEN_PROGS): $(BPFOBJ) +$(TEST_GEN_PROGS): test_stub.o $(BPFOBJ) -$(TEST_GEN_PROGS_EXTENDED): $(OUTPUT)/libbpf.a +$(TEST_GEN_PROGS_EXTENDED): test_stub.o $(OUTPUT)/libbpf.a $(OUTPUT)/test_dev_cgroup: cgroup_helpers.c $(OUTPUT)/test_skb_cgroup_id_user: cgroup_helpers.c @@ -174,7 +176,7 @@ $(ALU32_BUILD_DIR)/test_progs_32: test_progs.c $(OUTPUT)/libbpf.a\ $(ALU32_BUILD_DIR)/urandom_read $(CC) $(TEST_PROGS_CFLAGS) $(CFLAGS) \ -o $(ALU32_BUILD_DIR)/test_progs_32 \ - test_progs.c trace_helpers.c prog_tests/*.c \ + test_progs.c test_stub.c trace_helpers.c prog_tests/*.c \ $(OUTPUT)/libbpf.a $(LDLIBS) $(ALU32_BUILD_DIR)/test_progs_32: $(PROG_TESTS_H) diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c index 23b159d..2623d15 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c @@ -22,6 +22,7 @@ static int check_load(const char *file) attr.file = file; attr.prog_type = BPF_PROG_TYPE_SCHED_CLS; attr.log_level = 4; + attr.prog_flags = BPF_F_TEST_RND_HI32; err = bpf_prog_load_xattr(&attr, &obj, &prog_fd); bpf_object__close(obj); if (err) diff --git a/tools/testing/selftests/bpf/test_sock_addr.c b/tools/testing/selftests/bpf/test_sock_addr.c index 3f110ea..5d0c4f0 100644 --- a/tools/testing/selftests/bpf/test_sock_addr.c +++ b/tools/testing/selftests/bpf/test_sock_addr.c @@ -745,6 +745,7 @@ static int load_path(const struct sock_addr_test *test, const char *path) attr.file = path; attr.prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR; attr.expected_attach_type = test->expected_attach_type; + attr.prog_flags = BPF_F_TEST_RND_HI32; if (bpf_prog_load_xattr(&attr, &obj, &prog_fd)) { if (test->expected_result != LOAD_REJECT) diff --git a/tools/testing/selftests/bpf/test_sock_fields.c b/tools/testing/selftests/bpf/test_sock_fields.c index dcae7f6..f08c8ee 100644 --- a/tools/testing/selftests/bpf/test_sock_fields.c +++ b/tools/testing/selftests/bpf/test_sock_fields.c @@ -339,6 +339,7 @@ int main(int argc, char **argv) struct bpf_prog_load_attr attr = { .file = "test_sock_fields_kern.o", .prog_type = BPF_PROG_TYPE_CGROUP_SKB, + .prog_flags = BPF_F_TEST_RND_HI32, }; int cgroup_fd, egress_fd, ingress_fd, err; struct bpf_program *ingress_prog; diff --git a/tools/testing/selftests/bpf/test_socket_cookie.c b/tools/testing/selftests/bpf/test_socket_cookie.c index e51d637..cac8ee5 100644 --- a/tools/testing/selftests/bpf/test_socket_cookie.c +++ b/tools/testing/selftests/bpf/test_socket_cookie.c @@ -148,6 +148,7 @@ static int run_test(int cgfd) memset(&attr, 0, sizeof(attr)); attr.file = SOCKET_COOKIE_PROG; attr.prog_type = BPF_PROG_TYPE_UNSPEC; + attr.prog_flags = BPF_F_TEST_RND_HI32; err = bpf_prog_load_xattr(&attr, &pobj, &prog_fd); if (err) { diff --git a/tools/testing/selftests/bpf/test_stub.c b/tools/testing/selftests/bpf/test_stub.c new file mode 100644 index 0000000..84e81a8 --- /dev/null +++ b/tools/testing/selftests/bpf/test_stub.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +/* Copyright (C) 2019 Netronome Systems, Inc. */ + +#include +#include +#include + +int bpf_prog_test_load(const char *file, enum bpf_prog_type type, + struct bpf_object **pobj, int *prog_fd) +{ + struct bpf_prog_load_attr attr; + + memset(&attr, 0, sizeof(struct bpf_prog_load_attr)); + attr.file = file; + attr.prog_type = type; + attr.expected_attach_type = 0; + attr.prog_flags = BPF_F_TEST_RND_HI32; + + return bpf_prog_load_xattr(&attr, pobj, prog_fd); +} + +int bpf_test_load_program(enum bpf_prog_type type, const struct bpf_insn *insns, + size_t insns_cnt, const char *license, + __u32 kern_version, char *log_buf, + size_t log_buf_sz) +{ + struct bpf_load_program_attr load_attr; + + memset(&load_attr, 0, sizeof(struct bpf_load_program_attr)); + load_attr.prog_type = type; + load_attr.expected_attach_type = 0; + load_attr.name = NULL; + load_attr.insns = insns; + load_attr.insns_cnt = insns_cnt; + load_attr.license = license; + load_attr.kern_version = kern_version; + load_attr.prog_flags = BPF_F_TEST_RND_HI32; + + return bpf_load_program_xattr(&load_attr, log_buf, log_buf_sz); +} diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index e2ebcad..a5eacc8 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -161,7 +161,7 @@ static void bpf_fill_ld_abs_vlan_push_pop(struct bpf_test *self) goto loop; for (; i < len - 1; i++) - insn[i] = BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 0xbef); + insn[i] = BPF_ALU64_IMM(BPF_MOV, BPF_REG_0, 0xbef); insn[len - 1] = BPF_EXIT_INSN(); self->prog_len = len; } @@ -170,7 +170,7 @@ static void bpf_fill_jump_around_ld_abs(struct bpf_test *self) { struct bpf_insn *insn = self->fill_insns; /* jump range is limited to 16 bit. every ld_abs is replaced by 6 insns */ - unsigned int len = (1 << 15) / 6; + unsigned int len = (1 << 15) / 9; int i = 0; insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); @@ -783,7 +783,7 @@ static void do_test_single(struct bpf_test *test, bool unpriv, if (fixup_skips != skips) return; - pflags = 0; + pflags = BPF_F_TEST_RND_HI32; if (test->flags & F_LOAD_WITH_STRICT_ALIGNMENT) pflags |= BPF_F_STRICT_ALIGNMENT; if (test->flags & F_NEEDS_EFFICIENT_UNALIGNED_ACCESS) From patchwork Fri Apr 12 21:59:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084949 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="PWdoZ8gr"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJj33cjz9sN4 for ; Sat, 13 Apr 2019 08:00:21 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727290AbfDLWAU (ORCPT ); Fri, 12 Apr 2019 18:00:20 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:45842 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727263AbfDLWAS (ORCPT ); Fri, 12 Apr 2019 18:00:18 -0400 Received: by mail-wr1-f66.google.com with SMTP id s15so13696669wra.12 for ; Fri, 12 Apr 2019 15:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u7fZb3IEqFJRYe0oJUr+G1NSo0sv1IV8cJXsSnOcig0=; b=PWdoZ8grCfTD8nOp7OrDYEhnOKSrSKq87DzC6lV4PZ8CGpkSAHYiI1wIe1C3JkBD5l r/nyGaMlXT+FtS6c5QNz3xSEEsdw4azIsyUe8QPD01DyehXWpGfChrhO1zkKerNrhBD+ mFN3J2rA/4iLxwyjFQN7DhfwW4aZy09VIJZtbxDs+mJmpQWLFlmn3Xy3cNNPnE4T4g/u 0eBzkmGvMY/uX+HHOCTVA4I3A6Rf6V9dSD1up8Ux6fKBIFy9BSUzeTUTWQYwmMV25ncD qRtVpJ6oqHy5fXfgCGQnN0Kjca57lAZ6i5l8SH/4gujLeNn2k0s4u0kZho1oD5prgdx8 k4bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u7fZb3IEqFJRYe0oJUr+G1NSo0sv1IV8cJXsSnOcig0=; b=a1flVDFbi+ULPLifhEdy+FKXxdNzC3LP6tN033kGnibrPgF+5Gt0w4j6trCio07VrV QjSMDXlrxjmAx0mBRUGCm2uXbW/Q+sveUf9Nslbx/G8kgpy/nCHtA/gTlla/Xa5+Fv2t tzOP6I4CDiXZVQvERUm0jpBZOgKgeG19igLQalY5w61lw+Z46RQPWG22nZe3jrodJWLF itfYtlGx0EXaEFCnTuJQ44aRhFgvUi77rhIHR/6DI6G3z6xEnwGNsn/0SI2HCaY2LlhT ERj9+Kg0TtJhyE0NCrT7U9SXG/hIjOBg3Q41J4gmwtALYN18IWkX5LXm4Exo3fuOkYYP XmJg== X-Gm-Message-State: APjAAAWjIcxYXRes+qmv7odQoz9jaonf4uk7eU8W7IUNoeT32lPYcPyc J38wDEKynqR5hdeh8+66L1PjMQ== X-Google-Smtp-Source: APXvYqzHxMQKW6YRTv+3Xu/l3AfjHs7tRnsSD1QGXND72AwDdRKYWeWMYL92rWL+OckFRF3yNULcaA== X-Received: by 2002:a5d:4e87:: with SMTP id e7mr8265477wru.161.1555106416596; Fri, 12 Apr 2019 15:00:16 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:15 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Shubham Bansal Subject: [PATCH v3 bpf-next 13/19] arm: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:46 +0100 Message-Id: <1555106392-20117-14-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Cc: Shubham Bansal Signed-off-by: Jiong Wang --- arch/arm/net/bpf_jit_32.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index c8bfbbf..8cecd06 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -736,7 +736,8 @@ static inline void emit_a32_alu_r64(const bool is64, const s8 dst[], /* ALU operation */ emit_alu_r(rd[1], rs, true, false, op, ctx); - emit_a32_mov_i(rd[0], 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(rd[0], 0, ctx); } arm_bpf_put_reg64(dst, rd, ctx); @@ -758,8 +759,9 @@ static inline void emit_a32_mov_r64(const bool is64, const s8 dst[], struct jit_ctx *ctx) { if (!is64) { emit_a32_mov_r(dst_lo, src_lo, ctx); - /* Zero out high 4 bytes */ - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + /* Zero out high 4 bytes */ + emit_a32_mov_i(dst_hi, 0, ctx); } else if (__LINUX_ARM_ARCH__ < 6 && ctx->cpu_architecture < CPU_ARCH_ARMv5TE) { /* complete 8 byte move */ @@ -1438,7 +1440,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) } emit_udivmod(rd_lo, rd_lo, rt, ctx, BPF_OP(code)); arm_bpf_put_reg32(dst_lo, rd_lo, ctx); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; case BPF_ALU64 | BPF_DIV | BPF_K: case BPF_ALU64 | BPF_DIV | BPF_X: @@ -1453,7 +1456,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) return -EINVAL; if (imm) emit_a32_alu_i(dst_lo, imm, ctx, BPF_OP(code)); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; /* dst = dst << imm */ case BPF_ALU64 | BPF_LSH | BPF_K: @@ -1488,7 +1492,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) /* dst = ~dst */ case BPF_ALU | BPF_NEG: emit_a32_alu_i(dst_lo, 0, ctx, BPF_OP(code)); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; /* dst = ~dst (64 bit) */ case BPF_ALU64 | BPF_NEG: @@ -1838,6 +1843,11 @@ void bpf_jit_compile(struct bpf_prog *prog) /* Nothing to do here. We support Internal BPF. */ } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_prog *tmp, *orig_prog = prog; From patchwork Fri Apr 12 21:59:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084962 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="GzYIX3VK"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJz3h0Rz9s7T for ; Sat, 13 Apr 2019 08:00:35 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727301AbfDLWAe (ORCPT ); Fri, 12 Apr 2019 18:00:34 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:40037 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727277AbfDLWAT (ORCPT ); Fri, 12 Apr 2019 18:00:19 -0400 Received: by mail-wm1-f66.google.com with SMTP id z24so12734517wmi.5 for ; Fri, 12 Apr 2019 15:00:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FyyxR9Iqk6cAH0oBjCaJb0bcVV0NGMOOW5Oun5gFHdM=; b=GzYIX3VKctvBc2dtvJ16iqFpJneBC6lA4gUWtjc4rDtUOyQDQACrJmf/ffwJ/17EzV ryGcqB3E2HZ4BG8zvj+sP8N4BBxHxsD7pf3VYeMnAkK6wuMg/MbA2DflQixp+7kODyVM Bl6A55sxVt8IowrKxGEos7H1WUAvDnRH0uOy3wfbV1FwyRG7U8xfuU76ES0Xjtl6+QUu XyCAtbuprm0xyhRgICR9aahKf5z3vZ+21dWQVZND5Uz+bkUizLbOWRvFVeThJXcV9w8+ cOA1MRnRirYz0uHTiOxPVAS/gda1ySbVyrLG8TN2/WmHWCZFaiRPpvFU2wCIUz11Lvh5 FVCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FyyxR9Iqk6cAH0oBjCaJb0bcVV0NGMOOW5Oun5gFHdM=; b=jin7wFEGgJv5+AR7Nk0/mSb3h79tXNws0p8c5uK/gAqdy2s/4WDiGFVNWy9ag2FH6l ANI4FDYwi+PCNGKms063Bh8/kb4VjkJKEX6rpMMv2fy8Ch/kJTzT8QdhMggj88/KwS2t huaUheU3evwTps9CssFaoTi+Ri+AGcH41+H+TqXrRGmYMN/dYxjEw8979wqpKPX9t3+I vPCeLLpbwsPV4Gmt1RbJNSMjLRdGJyyYQRKdtl/PCj8p1MRSV2RiRFrbweMj5ZVecsHH bOQ2FGHWb8hX0wFZ9kYDZXkwKAfSFyc+CASKvpJNtnJF0kKYbOfNCICAKp2cRR0zPIp+ eaXA== X-Gm-Message-State: APjAAAWznzGE4qTFaXCs1Lfr9xZFvi4uil3yaBMYXGszSfoOWJvr635v tK0j96OpQoloPlwe1Src8N2euA== X-Google-Smtp-Source: APXvYqzC0PxQp0Q/y1dMtjmEK0vjYwZU7d77qu34HT2P4CZAC2TdvSATMtk7FNbgkx7ih4A0kwZilA== X-Received: by 2002:a1c:39d7:: with SMTP id g206mr11850480wma.99.1555106417615; Fri, 12 Apr 2019 15:00:17 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:16 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , "Naveen N . Rao" , Sandipan Das Subject: [PATCH v3 bpf-next 14/19] powerpc: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:47 +0100 Message-Id: <1555106392-20117-15-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: Naveen N. Rao Cc: Sandipan Das Signed-off-by: Jiong Wang --- arch/powerpc/net/bpf_jit_comp64.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 21a1dcd..d10621b 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -559,7 +559,7 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, bpf_alu32_trunc: /* Truncate to 32-bits */ - if (BPF_CLASS(code) == BPF_ALU) + if (BPF_CLASS(code) == BPF_ALU && fp->aux->no_verifier_zext) PPC_RLWINM(dst_reg, dst_reg, 0, 0, 31); break; @@ -1046,6 +1046,11 @@ struct powerpc64_jit_data { struct codegen_context ctx; }; +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) { u32 proglen; From patchwork Fri Apr 12 21:59:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084954 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="BoHHBDqS"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJs0kRSz9s71 for ; Sat, 13 Apr 2019 08:00:29 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727308AbfDLWAX (ORCPT ); Fri, 12 Apr 2019 18:00:23 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:33398 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727281AbfDLWAU (ORCPT ); Fri, 12 Apr 2019 18:00:20 -0400 Received: by mail-wr1-f67.google.com with SMTP id q1so13777398wrp.0 for ; Fri, 12 Apr 2019 15:00:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/wylRhv97waN1hLNQRjtw5HFd7aLKXwfbfkeZTfcGns=; b=BoHHBDqSo9hYv0bkI611gzs7drYaYY98UgiHOhpRkui7YtSrsec1ln6rGGvug6NAYQ sps7qnbUecCFq81vL5n50ALFCCJiNkmrxyc1zqjo1sdjBcVUP/1GCTUPW/cN+kfQlfvF ZvQ/6n3300/0iqhWAmFna7p1h3BCi7P2XeQarh79q393JlCnIGhsMs0sIKqhgMHgGoHk bvIhvXVOvVWANqv/YnBq7cik2mzFgnOnXf/Vr64Z1D+IpDi85/hdTPCdusBop+HQlHqB 2bF1sFqjrM+YRzaJMtdkK9q2Edjic9YBVKlQRARPululNxm/Y5gcXEGso2IxN+5lK/HY TmSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/wylRhv97waN1hLNQRjtw5HFd7aLKXwfbfkeZTfcGns=; b=H0C1Kd66J3z59YGj5IrNYpVnKvudo2q8s15JpN8bv072cMuMMmVLSqQZnBCRDEHEc8 v7NeWjSf9DeAeM77a69DMyvgnLMX02GBTqVOg8Y/sAFX7mWaF0HV2sGdxMMkUIv/tx3i kjwEqLOLLFmONhsNraSQ4Ci5/gMSGIO4+ixYZG9cO5nVug/YToGzZQ1RKGi29eBNM54i thT28jj1Cqz2+cnTiJvxVxylYFWeO8RAQFzbGc5iwRrJcuZstZaMvveCRTm27O8IcIuF Th9p4X2Nqt0Y8jJ4b71NX16op9KpGThyk8BTMaDa15SsbJ5l2Ex6orp6qIn4VLhZoMC7 ziCw== X-Gm-Message-State: APjAAAVXieqNMNjKTfozC6OR7Z3nmDZQQUBkqhDKW4JQpDQ4w7noa2TC 5PWMXzIXz/NBLzXWR5UwwVY7Fg== X-Google-Smtp-Source: APXvYqzrxDF22KbtqLIyFMfNsKL/hMb0KNXMx9QPeLg6JA5ur1aUZwyF1It0qkQSPIBfhk5gQpHUwg== X-Received: by 2002:adf:ea88:: with SMTP id s8mr12063138wrm.300.1555106418532; Fri, 12 Apr 2019 15:00:18 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:17 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Martin Schwidefsky , Heiko Carstens Subject: [PATCH v3 bpf-next 15/19] s390: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:48 +0100 Message-Id: <1555106392-20117-16-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Cc: Martin Schwidefsky Cc: Heiko Carstens Signed-off-by: Jiong Wang --- arch/s390/net/bpf_jit_comp.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 51dd026..59592d7 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -299,9 +299,11 @@ static inline void reg_set_seen(struct bpf_jit *jit, u32 b1) #define EMIT_ZERO(b1) \ ({ \ - /* llgfr %dst,%dst (zero extend to 64 bit) */ \ - EMIT4(0xb9160000, b1, b1); \ - REG_SET_SEEN(b1); \ + if (fp->aux->no_verifier_zext) { \ + /* llgfr %dst,%dst (zero extend to 64 bit) */ \ + EMIT4(0xb9160000, b1, b1); \ + REG_SET_SEEN(b1); \ + } \ }) /* @@ -1282,6 +1284,11 @@ static int bpf_jit_prog(struct bpf_jit *jit, struct bpf_prog *fp) return 0; } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + /* * Compile eBPF program "fp" */ From patchwork Fri Apr 12 21:59:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084950 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="HwKdVsjZ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJl044yz9s7T for ; Sat, 13 Apr 2019 08:00:23 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727298AbfDLWAW (ORCPT ); Fri, 12 Apr 2019 18:00:22 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:37341 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727287AbfDLWAU (ORCPT ); Fri, 12 Apr 2019 18:00:20 -0400 Received: by mail-wm1-f66.google.com with SMTP id v14so12644705wmf.2 for ; Fri, 12 Apr 2019 15:00:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Pi+jMPgfzj2fBkAwzCkG0vyOGnD4cZfJGlYpVzkjSMY=; b=HwKdVsjZ+lKkcdRiB7Sd+PAgUvTf9Ik/ylIbnc9ioyPXpFQZ58fK0VEHrj0HC4Gh9M 3V7ZhulYxqUA8PKA3vgJhhMKRYGYWfde06D1ZdY9bfn1b2L2bAYhN+300Lga1fwtm/8h eDD5I9HluoqXq8ne14lJ/CRo+VVfA+nR5hDYXvMmGtsEAHzFmY4x0GCB/KZRYz2QD65M rAQUByulZq6V9Q/PrQ4fesAP81q/MZOLC10OCUNi9bMd01ZCoF3+UpQs/SNlTP0iwcgu 9hD+d+M35MFHf5vCXycIWDvrjKGHvawTnr/uZR7hevXcPbpm15rRpw7FRzxX5J4qkY3h Q2dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Pi+jMPgfzj2fBkAwzCkG0vyOGnD4cZfJGlYpVzkjSMY=; b=mJUGo2sHoFNzJkDp5y0qb5bzZDNftuwxSx6T7LC/8bwlFJ1PqOSa25jfqvogWg2xT5 jPWnd40zjiouw3Fe4b6sOzQBU0O3PG8b4M6q6xMrBCAg4p92riRF8IJN2q0/U0KSti2D nGkEuelJUusgIaWrhRoLgLYIkkhlZ6RY+zzX5ni8rMgpd0e3JQVTLjGv4qePwLZBuYgm WCkrsSiLnePSu7qtSaxkY3571PKPmOsuA0bmwWUv7JPzqW6KNC66FTLjUcWKRbgnpRkR Td7uJBeFJZVR0JfgEvW4U8ApopJWBC/luEPicwt8/NMyJ5DH6w/A47sIiosw8dGWvFKi dG/Q== X-Gm-Message-State: APjAAAWPL3p6kyWdZ7RJsauBcH0Nb84vJ4doIekQUrvDP+4g77JdG9d4 tICBFaRIVsLYSHiA0ruZX6pITQ== X-Google-Smtp-Source: APXvYqyLyRHzjd9TagkWVLihgtpSoDoJ9UIdJAxZAMG91CJi5vc5zuovv2Y90ro7onQq+SuTM/zaRw== X-Received: by 2002:a1c:988d:: with SMTP id a135mr13692229wme.24.1555106419277; Fri, 12 Apr 2019 15:00:19 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:18 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , "David S . Miller" Subject: [PATCH v3 bpf-next 16/19] sparc: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:49 +0100 Message-Id: <1555106392-20117-17-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: David S. Miller Signed-off-by: Jiong Wang --- arch/sparc/net/bpf_jit_comp_64.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c index 65428e7..e58f84e 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -1144,7 +1144,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) break; do_alu32_trunc: - if (BPF_CLASS(code) == BPF_ALU) + if (BPF_CLASS(code) == BPF_ALU && + ctx->prog->aux->no_verifier_zext) emit_alu_K(SRL, dst, 0, ctx); break; @@ -1432,6 +1433,11 @@ static void jit_fill_hole(void *area, unsigned int size) *ptr++ = 0x91d02005; /* ta 5 */ } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct sparc64_jit_data { struct bpf_binary_header *header; u8 *image; From patchwork Fri Apr 12 21:59:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084951 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="CQUh5itm"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJp0dlsz9s71 for ; Sat, 13 Apr 2019 08:00:26 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727315AbfDLWAY (ORCPT ); Fri, 12 Apr 2019 18:00:24 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:42372 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727291AbfDLWAW (ORCPT ); Fri, 12 Apr 2019 18:00:22 -0400 Received: by mail-wr1-f68.google.com with SMTP id g3so13700424wrx.9 for ; Fri, 12 Apr 2019 15:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=esldtJjtJ7LjAyiBdO+8AWliqTz0U0S4jG86huzM9ss=; b=CQUh5itmka6l4NBt2SuRpb99diz0zjk/q1wHkLpszCUeugGhixpxnG8h7IZdBpDvbH kpS4/i3kXn7oU6X61IkDGHZxhneigrWtvfaGkGjyqQVH6/CT3UA3p2m1r73p7pz+pu6d xwl6Y7giTmDeCmaO5v4SeYrkCUV4gWUdFgcY5/b+Wf6dViI5ufBMXFJga/eHm10uRuX/ oE0p6J1xzMl5HDjGu6fOTv7cF8WgUYwNjFRdCM36WCmCyFSi6wslxz+DVt49rf5XnLbZ 5M+xIuphkzSSFEx82nhOZOd9sps/i21/AeR6iXy5VTHIx+8mqGSqqXaexIFfMGcSfLTw 8UNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=esldtJjtJ7LjAyiBdO+8AWliqTz0U0S4jG86huzM9ss=; b=ZO/VpTp3XXptOBlMGUlEaVXqSjTJkv3AsyWmwye04itLo3KZ1GvaEvcVO5gjZNE8PZ Jx1CNcK42++SdSZuIKl7lrY7r2RfrNDOkoniKJaWKdhMpivYmHmri1CXwYg6GmmlyWLP JB/GxIADPqywDckq+dSmxN4UupuRUUbDm3bh7MGfZoIx4c4dKeU2neYNm9IBB0rWObKW 9nZ74CWTtuc9f+yE0xAe/0zx+vt1CgxaDGiT0JKFWOKyF28If07AlwMVkYfSkwYnCLin +o66gQiEgqlyAUix2n7tdEB6c5181SDHrJYYbkA4Da078oPCH3mDQ53Wk8wGVop/PibJ Meig== X-Gm-Message-State: APjAAAUgkgctwXyLXQh3Cd2ztOdBl8iTUYQ8+NNEAzn5l4adN1qBh3sf ztzb0RtYv+bJA6Je8f5r9bf8Nw== X-Google-Smtp-Source: APXvYqwYOCRhHutxcmr7fm9xZcCjKXgtAfEYV/7ujY+qZxy7BZyH9Gmx4RRIUWarDolt3vhNGYVuEQ== X-Received: by 2002:a5d:5310:: with SMTP id e16mr16430717wrv.142.1555106420362; Fri, 12 Apr 2019 15:00:20 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:19 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Wang YanQing Subject: [PATCH v3 bpf-next 17/19] x32: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:50 +0100 Message-Id: <1555106392-20117-18-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Cc: Wang YanQing Signed-off-by: Jiong Wang --- arch/x86/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 0d9cdff..8c6cf22 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -567,7 +567,7 @@ static inline void emit_ia32_alu_r(const bool is64, const bool hi, const u8 op, static inline void emit_ia32_alu_r64(const bool is64, const u8 op, const u8 dst[], const u8 src[], bool dstk, bool sstk, - u8 **pprog) + u8 **pprog, const struct bpf_prog_aux *aux) { u8 *prog = *pprog; @@ -575,7 +575,7 @@ static inline void emit_ia32_alu_r64(const bool is64, const u8 op, if (is64) emit_ia32_alu_r(is64, true, op, dst_hi, src_hi, dstk, sstk, &prog); - else + else if (aux->no_verifier_zext) emit_ia32_mov_i(dst_hi, 0, dstk, &prog); *pprog = prog; } @@ -666,7 +666,8 @@ static inline void emit_ia32_alu_i(const bool is64, const bool hi, const u8 op, /* ALU operation (64 bit) */ static inline void emit_ia32_alu_i64(const bool is64, const u8 op, const u8 dst[], const u32 val, - bool dstk, u8 **pprog) + bool dstk, u8 **pprog, + const struct bpf_prog_aux *aux) { u8 *prog = *pprog; u32 hi = 0; @@ -677,7 +678,7 @@ static inline void emit_ia32_alu_i64(const bool is64, const u8 op, emit_ia32_alu_i(is64, false, op, dst_lo, val, dstk, &prog); if (is64) emit_ia32_alu_i(is64, true, op, dst_hi, hi, dstk, &prog); - else + else if (aux->no_verifier_zext) emit_ia32_mov_i(dst_hi, 0, dstk, &prog); *pprog = prog; @@ -1690,11 +1691,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, switch (BPF_SRC(code)) { case BPF_X: emit_ia32_alu_r64(is64, BPF_OP(code), dst, - src, dstk, sstk, &prog); + src, dstk, sstk, &prog, + bpf_prog->aux); break; case BPF_K: emit_ia32_alu_i64(is64, BPF_OP(code), dst, - imm32, dstk, &prog); + imm32, dstk, &prog, + bpf_prog->aux); break; } break; @@ -1713,7 +1716,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, false, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; case BPF_ALU | BPF_LSH | BPF_X: case BPF_ALU | BPF_RSH | BPF_X: @@ -1733,7 +1737,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; /* dst = dst / src(imm) */ /* dst = dst % src(imm) */ @@ -1755,7 +1760,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; case BPF_ALU64 | BPF_DIV | BPF_K: case BPF_ALU64 | BPF_DIV | BPF_X: @@ -1772,7 +1778,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); emit_ia32_shift_r(BPF_OP(code), dst_lo, IA32_ECX, dstk, false, &prog); - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; /* dst = dst << imm */ case BPF_ALU64 | BPF_LSH | BPF_K: @@ -2367,6 +2374,11 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, return proglen; } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_binary_header *header = NULL; From patchwork Fri Apr 12 21:59:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084956 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="EMfVPShx"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJv6fjqz9s7T for ; Sat, 13 Apr 2019 08:00:31 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727311AbfDLWAb (ORCPT ); Fri, 12 Apr 2019 18:00:31 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:36169 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727300AbfDLWAX (ORCPT ); Fri, 12 Apr 2019 18:00:23 -0400 Received: by mail-wr1-f65.google.com with SMTP id y13so13761476wrd.3 for ; Fri, 12 Apr 2019 15:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pNHJxDIKLsiKxKEQK4Ngr2CSz1DZ9y91uELS2TzjMyw=; b=EMfVPShx+E699tBHUlWyNGadB9lysg0dcgBrpGLzcJlqr5zSQG/ndyb4zvd+/rMQST SGoHHFzwycp75ZHwdOrVGtIB6oNrSauLYowzOBzLQnSiPCjZP2DjDwA2EDTYE//4HlQ1 73BSefIPWFIZUVPvI9WIBQlYyGB28LIKCa+b4A8m2DDHj4hw2nfvyPUcz74Exg9cbEX+ fsN46mFciva5ZqFHpvojbp2nQxlTceYH55NqzDEYIDZxKov3ynNoT/kLAbG0/8buqPtw ttgDo9EwQ7ZAW/baaqDrAUQOXBM6g+qtWbBIfpDKnLjyH1hhxB/YaQLmbumEnLqjL92X mGpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pNHJxDIKLsiKxKEQK4Ngr2CSz1DZ9y91uELS2TzjMyw=; b=oDog87bIIBLKJ5k8q/9u8jfVleruIQ5jaVQmKprvZq/BUaMz8UUXPgzvqQsSTcEm+Z 043hSCves2/1CDu73ELCROWrzi0AqR9vHfzv8MwdwoCwVRTmoD8UZYlSVOB+hcTsv1G3 pEsRIP2N9v0p3eY7Fuifw6EIulbfIGqmFXQ8YZxKv5Wtmj9db7DM9ePFsNuU+8HqFMEo QtGKn0l/uqa9KaydH8XcKLbrkP0GztFv2B4e0vjD+AkUy0+vW2gTdGEHzi7C+aM2zHdD Qkr6kc0D43XzQDXhr4JqdmSdo7bOPIp3gxIlLSAVeS4qaqYkVIuTMteuPB6XjbXTJIsv sO7A== X-Gm-Message-State: APjAAAWhyAfrG5mfO16YpgPbXgy87DbZyZo70ejAfXedlu/vNUDuSLkt 5BtHPq0Y0+lAT6SuXoF0fCMong== X-Google-Smtp-Source: APXvYqxrA35/+k8SBLu8Kmb6jgsSMvZw5NiiAHLCSavGjxP0E+CjzosStqeUG+oU2bKuBmUgBMdA5g== X-Received: by 2002:a5d:530f:: with SMTP id e15mr39994616wrv.19.1555106421626; Fri, 12 Apr 2019 15:00:21 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:20 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= Subject: [PATCH v3 bpf-next 18/19] riscv: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:51 +0100 Message-Id: <1555106392-20117-19-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org CC: Björn Töpel Signed-off-by: Jiong Wang --- arch/riscv/net/bpf_jit_comp.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 80b12aa..9cba262 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -731,6 +731,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, { bool is64 = BPF_CLASS(insn->code) == BPF_ALU64 || BPF_CLASS(insn->code) == BPF_JMP; + struct bpf_prog_aux *aux = ctx->prog->aux; int rvoff, i = insn - ctx->prog->insnsi; u8 rd = -1, rs = -1, code = insn->code; s16 off = insn->off; @@ -743,7 +744,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MOV | BPF_X: case BPF_ALU64 | BPF_MOV | BPF_X: emit(is64 ? rv_addi(rd, rs, 0) : rv_addiw(rd, rs, 0), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; @@ -771,19 +772,19 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MUL | BPF_X: case BPF_ALU64 | BPF_MUL | BPF_X: emit(is64 ? rv_mul(rd, rd, rs) : rv_mulw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_DIV | BPF_X: case BPF_ALU64 | BPF_DIV | BPF_X: emit(is64 ? rv_divu(rd, rd, rs) : rv_divuw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MOD | BPF_X: case BPF_ALU64 | BPF_MOD | BPF_X: emit(is64 ? rv_remu(rd, rd, rs) : rv_remuw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_LSH | BPF_X: @@ -867,7 +868,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MOV | BPF_K: case BPF_ALU64 | BPF_MOV | BPF_K: emit_imm(rd, imm, ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; @@ -882,7 +883,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit(is64 ? rv_add(rd, rd, RV_REG_T1) : rv_addw(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_SUB | BPF_K: @@ -895,7 +896,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit(is64 ? rv_sub(rd, rd, RV_REG_T1) : rv_subw(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_AND | BPF_K: @@ -906,7 +907,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_and(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_OR | BPF_K: @@ -917,7 +918,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_or(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_XOR | BPF_K: @@ -928,7 +929,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_xor(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MUL | BPF_K: @@ -936,7 +937,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_mul(rd, rd, RV_REG_T1) : rv_mulw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_DIV | BPF_K: @@ -944,7 +945,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_divu(rd, rd, RV_REG_T1) : rv_divuw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MOD | BPF_K: @@ -952,7 +953,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_remu(rd, rd, RV_REG_T1) : rv_remuw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_LSH | BPF_K: @@ -1503,6 +1504,11 @@ static void bpf_flush_icache(void *start, void *end) flush_icache_range((unsigned long)start, (unsigned long)end); } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { bool tmp_blinded = false, extra_pass = false; From patchwork Fri Apr 12 21:59:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1084952 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="mWAOoTmJ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44gsJq56jkz9s71 for ; Sat, 13 Apr 2019 08:00:27 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727319AbfDLWA0 (ORCPT ); Fri, 12 Apr 2019 18:00:26 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43826 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727311AbfDLWAZ (ORCPT ); Fri, 12 Apr 2019 18:00:25 -0400 Received: by mail-wr1-f68.google.com with SMTP id k17so9058019wrx.10 for ; Fri, 12 Apr 2019 15:00:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qxA1L/gVWESQpZ3Mokilcchz/Sf49i0tM5B88NgcAXI=; b=mWAOoTmJIJ3brFmu7XsdeQMJiLR7G8a1ltNkL2ny0Fk9bkXlV/WDM93PS/G5JEPODW 0z+Pcaau6UoUR0iPNd3Eo8wSPyhq6g+dzuTZpZqBwBWogV2JJVTVIeSWgjFowyBoXcAJ tByuXnabCGkPLcrb55EMcW26O/UIW4a6Oko5ZnxFKG8be+Ah3vPiLD0pw8938ZPE6LC0 npVulra5APo5d1E5/W/1qpbu55/UKl0BcZvv/vNL5jmZ/dbsKEjYHkbBx0Cyy/5dJt1/ YiXd9Lfz3aR28axjONwOG33qv80BM1Vbh97vTS93NRQuGaSYjILI20oVOkEo7mtLM6zV x9GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qxA1L/gVWESQpZ3Mokilcchz/Sf49i0tM5B88NgcAXI=; b=ZgPuCtuk9l/85kTgDAAMG0LjJn8ptaDBfHi0NxYFv828bYt8td8VvuoSgG4DwP7iOY 39hsHnMwra82ubKoLaYes3vG73HXICXysGi5O/dV3CuYtRBy8sxZoKv7ySkHCGzcGmPT hHjsQdSLKV6OOgoySpLD17KJQ8CuCtu5e5Aa13Er/xwYv/fD3lS6tm2RiV/ScIojZO5l CxRUyhN2YN46Tm/WcPd4K3RhkdzwV4P7HQdnr3Vj+EbXVd+8GXnW2F+pbJvD1G9udzcc Wcf9wmYjk8Em8NHsZPND3YRdfdi0siuVFRvp7/J5Y6gy4RfA7R4CVER/qY8Upkm/F+gu /2Ng== X-Gm-Message-State: APjAAAUWhO1g0GowHYm89Qkl4tSRHywP2+fMFToMB6/5/FbKS8D6eGYH 9zoRKmiMbWdEUHgFO5OZ5Rs2Hw== X-Google-Smtp-Source: APXvYqwcbwmWkr2NF9wyb8ZL4+mG0jVpfnhtXRQMWTdOlOnXU5WrZBtZRv/uRCwVZE9LsI9BISP+KQ== X-Received: by 2002:adf:fe41:: with SMTP id m1mr38306960wrs.220.1555106423068; Fri, 12 Apr 2019 15:00:23 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id f1sm8490764wml.28.2019.04.12.15.00.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Apr 2019 15:00:22 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH v3 bpf-next 19/19] nfp: bpf: eliminate zero extension code-gen Date: Fri, 12 Apr 2019 22:59:52 +0100 Message-Id: <1555106392-20117-20-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> References: <1555106392-20117-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch eliminate zero extension code-gen for instructions except load/store when possible. Elimination for load/store will be supported in up coming patches. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- drivers/net/ethernet/netronome/nfp/bpf/jit.c | 119 +++++++++++++--------- drivers/net/ethernet/netronome/nfp/bpf/main.h | 2 + drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 12 +++ 3 files changed, 83 insertions(+), 50 deletions(-) diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c index f272247..eb30c52 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c @@ -612,6 +612,13 @@ static void wrp_immed(struct nfp_prog *nfp_prog, swreg dst, u32 imm) } static void +wrp_zext(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst) +{ + if (meta->flags & FLAG_INSN_DO_ZEXT) + wrp_immed(nfp_prog, reg_both(dst + 1), 0); +} + +static void wrp_immed_relo(struct nfp_prog *nfp_prog, swreg dst, u32 imm, enum nfp_relo_type relo) { @@ -847,7 +854,8 @@ static int nfp_cpp_memcpy(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) } static int -data_ld(struct nfp_prog *nfp_prog, swreg offset, u8 dst_gpr, int size) +data_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, swreg offset, + u8 dst_gpr, int size) { unsigned int i; u16 shift, sz; @@ -870,14 +878,15 @@ data_ld(struct nfp_prog *nfp_prog, swreg offset, u8 dst_gpr, int size) wrp_mov(nfp_prog, reg_both(dst_gpr + i), reg_xfer(i)); if (i < 2) - wrp_immed(nfp_prog, reg_both(dst_gpr + 1), 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } static int -data_ld_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, - swreg lreg, swreg rreg, int size, enum cmd_mode mode) +data_ld_host_order(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 dst_gpr, swreg lreg, swreg rreg, int size, + enum cmd_mode mode) { unsigned int i; u8 mask, sz; @@ -900,33 +909,34 @@ data_ld_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, wrp_mov(nfp_prog, reg_both(dst_gpr + i), reg_xfer(i)); if (i < 2) - wrp_immed(nfp_prog, reg_both(dst_gpr + 1), 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } static int -data_ld_host_order_addr32(struct nfp_prog *nfp_prog, u8 src_gpr, swreg offset, - u8 dst_gpr, u8 size) +data_ld_host_order_addr32(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 src_gpr, swreg offset, u8 dst_gpr, u8 size) { - return data_ld_host_order(nfp_prog, dst_gpr, reg_a(src_gpr), offset, - size, CMD_MODE_32b); + return data_ld_host_order(nfp_prog, meta, dst_gpr, reg_a(src_gpr), + offset, size, CMD_MODE_32b); } static int -data_ld_host_order_addr40(struct nfp_prog *nfp_prog, u8 src_gpr, swreg offset, - u8 dst_gpr, u8 size) +data_ld_host_order_addr40(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 src_gpr, swreg offset, u8 dst_gpr, u8 size) { swreg rega, regb; addr40_offset(nfp_prog, src_gpr, offset, ®a, ®b); - return data_ld_host_order(nfp_prog, dst_gpr, rega, regb, + return data_ld_host_order(nfp_prog, meta, dst_gpr, rega, regb, size, CMD_MODE_40b_BA); } static int -construct_data_ind_ld(struct nfp_prog *nfp_prog, u16 offset, u16 src, u8 size) +construct_data_ind_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u16 offset, u16 src, u8 size) { swreg tmp_reg; @@ -942,10 +952,12 @@ construct_data_ind_ld(struct nfp_prog *nfp_prog, u16 offset, u16 src, u8 size) emit_br_relo(nfp_prog, BR_BLO, BR_OFF_RELO, 0, RELO_BR_GO_ABORT); /* Load data */ - return data_ld(nfp_prog, imm_b(nfp_prog), 0, size); + return data_ld(nfp_prog, meta, imm_b(nfp_prog), 0, size); } -static int construct_data_ld(struct nfp_prog *nfp_prog, u16 offset, u8 size) +static int +construct_data_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u16 offset, u8 size) { swreg tmp_reg; @@ -956,7 +968,7 @@ static int construct_data_ld(struct nfp_prog *nfp_prog, u16 offset, u8 size) /* Load data */ tmp_reg = re_load_imm_any(nfp_prog, offset, imm_b(nfp_prog)); - return data_ld(nfp_prog, tmp_reg, 0, size); + return data_ld(nfp_prog, meta, tmp_reg, 0, size); } static int @@ -1193,7 +1205,7 @@ mem_op_stack(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, } if (clr_gpr && size < 8) - wrp_immed(nfp_prog, reg_both(gpr + 1), 0); + wrp_zext(nfp_prog, meta, gpr); while (size) { u32 slice_end; @@ -1294,9 +1306,10 @@ wrp_alu32_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, enum alu_op alu_op) { const struct bpf_insn *insn = &meta->insn; + u8 dst = insn->dst_reg * 2; - wrp_alu_imm(nfp_prog, insn->dst_reg * 2, alu_op, insn->imm); - wrp_immed(nfp_prog, reg_both(insn->dst_reg * 2 + 1), 0); + wrp_alu_imm(nfp_prog, dst, alu_op, insn->imm); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -1308,7 +1321,7 @@ wrp_alu32_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst = meta->insn.dst_reg * 2, src = meta->insn.src_reg * 2; emit_alu(nfp_prog, reg_both(dst), reg_a(dst), alu_op, reg_b(src)); - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2385,12 +2398,14 @@ static int neg_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) u8 dst = meta->insn.dst_reg * 2; emit_alu(nfp_prog, reg_both(dst), reg_imm(0), ALU_OP_SUB, reg_b(dst)); - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } -static int __ashr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__ashr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) { /* Set signedness bit (MSB of result). */ @@ -2399,7 +2414,7 @@ static int __ashr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_ASHR, reg_b(dst), SHF_SC_R_SHF, shift_amt); } - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2414,7 +2429,7 @@ static int ashr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __ashr_imm(nfp_prog, dst, umin); + return __ashr_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; /* NOTE: the first insn will set both indirect shift amount (source A) @@ -2423,7 +2438,7 @@ static int ashr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) emit_alu(nfp_prog, reg_none(), reg_a(src), ALU_OP_OR, reg_b(dst)); emit_shf_indir(nfp_prog, reg_both(dst), reg_none(), SHF_OP_ASHR, reg_b(dst), SHF_SC_R_SHF); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2433,15 +2448,17 @@ static int ashr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __ashr_imm(nfp_prog, dst, insn->imm); + return __ashr_imm(nfp_prog, meta, dst, insn->imm); } -static int __shr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__shr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_R_SHF, shift_amt); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2450,7 +2467,7 @@ static int shr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __shr_imm(nfp_prog, dst, insn->imm); + return __shr_imm(nfp_prog, meta, dst, insn->imm); } static int shr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) @@ -2463,22 +2480,24 @@ static int shr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __shr_imm(nfp_prog, dst, umin); + return __shr_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; emit_alu(nfp_prog, reg_none(), reg_a(src), ALU_OP_OR, reg_imm(0)); emit_shf_indir(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_R_SHF); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } -static int __shl_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__shl_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_L_SHF, shift_amt); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2487,7 +2506,7 @@ static int shl_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __shl_imm(nfp_prog, dst, insn->imm); + return __shl_imm(nfp_prog, meta, dst, insn->imm); } static int shl_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) @@ -2500,11 +2519,11 @@ static int shl_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __shl_imm(nfp_prog, dst, umin); + return __shl_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; shl_reg64_lt32_low(nfp_prog, dst, src); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2566,34 +2585,34 @@ static int imm_ld8(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) static int data_ld1(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 1); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 1); } static int data_ld2(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 2); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 2); } static int data_ld4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 4); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 4); } static int data_ind_ld1(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 1); } static int data_ind_ld2(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 2); } static int data_ind_ld4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 4); } @@ -2632,7 +2651,7 @@ static int mem_ldx_skb(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, return -EOPNOTSUPP; } - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2658,7 +2677,7 @@ static int mem_ldx_xdp(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, return -EOPNOTSUPP; } - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2671,7 +2690,7 @@ mem_ldx_data(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, tmp_reg = re_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog)); - return data_ld_host_order_addr32(nfp_prog, meta->insn.src_reg * 2, + return data_ld_host_order_addr32(nfp_prog, meta, meta->insn.src_reg * 2, tmp_reg, meta->insn.dst_reg * 2, size); } @@ -2683,7 +2702,7 @@ mem_ldx_emem(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, tmp_reg = re_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog)); - return data_ld_host_order_addr40(nfp_prog, meta->insn.src_reg * 2, + return data_ld_host_order_addr40(nfp_prog, meta, meta->insn.src_reg * 2, tmp_reg, meta->insn.dst_reg * 2, size); } @@ -2744,7 +2763,7 @@ mem_ldx_data_from_pktcache_unaligned(struct nfp_prog *nfp_prog, wrp_reg_subpart(nfp_prog, dst_lo, src_lo, len_lo, off); if (!len_mid) { - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } @@ -2752,7 +2771,7 @@ mem_ldx_data_from_pktcache_unaligned(struct nfp_prog *nfp_prog, if (size <= REG_WIDTH) { wrp_reg_or_subpart(nfp_prog, dst_lo, src_mid, len_mid, len_lo); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else { swreg src_hi = reg_xfer(idx + 2); @@ -2783,10 +2802,10 @@ mem_ldx_data_from_pktcache_aligned(struct nfp_prog *nfp_prog, if (size < REG_WIDTH) { wrp_reg_subpart(nfp_prog, dst_lo, src_lo, size, 0); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else if (size == REG_WIDTH) { wrp_mov(nfp_prog, dst_lo, src_lo); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else { swreg src_hi = reg_xfer(idx + 1); diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h index b25a482..7369bdf 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/main.h +++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h @@ -249,6 +249,8 @@ struct nfp_bpf_reg_state { #define FLAG_INSN_SKIP_PREC_DEPENDENT BIT(4) /* Instruction is optimized by the verifier */ #define FLAG_INSN_SKIP_VERIFIER_OPT BIT(5) +/* Instruction needs to zero extend to high 32-bit */ +#define FLAG_INSN_DO_ZEXT BIT(6) #define FLAG_INSN_SKIP_MASK (FLAG_INSN_SKIP_NOOP | \ FLAG_INSN_SKIP_PREC_DEPENDENT | \ diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c index 36f56eb..e92ee51 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c @@ -744,6 +744,17 @@ static unsigned int nfp_bpf_get_stack_usage(struct nfp_prog *nfp_prog) goto continue_subprog; } +static void nfp_bpf_insn_flag_zext(struct nfp_prog *nfp_prog, + struct bpf_insn_aux_data *aux) +{ + struct nfp_insn_meta *meta; + + list_for_each_entry(meta, &nfp_prog->insns, l) { + if (aux[meta->n].zext_dst) + meta->flags |= FLAG_INSN_DO_ZEXT; + } +} + int nfp_bpf_finalize(struct bpf_verifier_env *env) { struct bpf_subprog_info *info; @@ -784,6 +795,7 @@ int nfp_bpf_finalize(struct bpf_verifier_env *env) return -EOPNOTSUPP; } + nfp_bpf_insn_flag_zext(nfp_prog, env->insn_aux_data); return 0; }