From patchwork Tue Nov 19 19:30:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197599 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="NHuwhfJM"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbX94C4Mz9sPZ for ; Wed, 20 Nov 2019 06:30:49 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727254AbfKSTas (ORCPT ); Tue, 19 Nov 2019 14:30:48 -0500 Received: from mail-pf1-f201.google.com ([209.85.210.201]:45620 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727217AbfKSTaq (ORCPT ); Tue, 19 Nov 2019 14:30:46 -0500 Received: by mail-pf1-f201.google.com with SMTP id a14so17501479pfr.12 for ; Tue, 19 Nov 2019 11:30:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SLNOUoddbyuUDBBjWCQQAd7kt0eAjd+1Ui6xbGCwfdw=; b=NHuwhfJMeBFzBFIpGu7GHUr65c5F4D0GPRXPJzg/r3mQTrUz+H+/PU8657f20fsXY6 avvTS/hywe/PyuN2IXCiAQzPNZHvfs58l6nkyFnMQSFgaY0mcwNptAWvAh0nRJsOHysf h0qXYGbo9SHSjuZp6V072AF6sJQl92k7iYOQnyseO9zklxSTLXkBaf6a5bM7qwkjz9Zl UoLds0n69ycgeRwj8u4lsb5/UoD5wkIsKpleJCu4B5I28c/jEmbDiU/sqGiPMJt5GedE Ti044bvd6VBc0o064XWDCWO83AVJJOYyAEqtfRjn5WYJn8lfwTtZVxxQAz/oGxjsKvtm wiCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SLNOUoddbyuUDBBjWCQQAd7kt0eAjd+1Ui6xbGCwfdw=; b=nupkSXLpW47sk3r4O2B2dTZ2qdzqKSJSnEstQl0leo7Tmd2QrlIaY+QxuLa3E9lu9D R2hxVpbq2PfdqyVSMK9Nyq399n7bPDLWMKYAJElwLRYJ9boEOx5L64/DvmGJ6gnI3RSO Fzz7V32pXkCbFtvia1mB295fdwaAmpET2zpLO+WKw2aSHLqjJsuBdnzoTm+aGA+WOamT 3eMcPKvG84Dvo4ub25IsKyfvMoWHVNRZMZpX11MF82TWvwoBg/bMcArhU/uxfRfg82dk M1d95aXwAMEWuFk/ezSfAHxQ34yIW86Uuqm41+X3OXgie3oDyUKA0rHmiHkUUsHe60yb 1cJw== X-Gm-Message-State: APjAAAWXXgVHcGACam5yLEy9y2Ys0Sn6LrjV2JJ9IjRh0TolZ7LnohJm HtaF4M51FEu3tnfMI6XaN4lh1LXCc8WZ X-Google-Smtp-Source: APXvYqz0CgjGXUAWmPr8wA/UlSDAjeWyRHnX9g956izLy95vulGwB8v2eRZLghNZ4UbVw4ZLAjPIRfOagPgL X-Received: by 2002:a63:df09:: with SMTP id u9mr300681pgg.20.1574191845498; Tue, 19 Nov 2019 11:30:45 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:28 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-2-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 1/9] bpf: add bpf_map_{value_size, update_value, map_copy_value} functions From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit moves reusable code from map_lookup_elem and map_update_elem to avoid code duplication in kernel/bpf/syscall.c. Signed-off-by: Brian Vazquez Acked-by: John Fastabend --- kernel/bpf/syscall.c | 271 ++++++++++++++++++++++++------------------- 1 file changed, 151 insertions(+), 120 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index c88c815c2154d..cc714c9d5b4cc 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -127,6 +127,153 @@ static struct bpf_map *find_and_alloc_map(union bpf_attr *attr) return map; } +static u32 bpf_map_value_size(struct bpf_map *map) +{ + if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || + map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || + map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY || + map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) + return round_up(map->value_size, 8) * num_possible_cpus(); + else if (IS_FD_MAP(map)) + return sizeof(u32); + else + return map->value_size; +} + +static void maybe_wait_bpf_programs(struct bpf_map *map) +{ + /* Wait for any running BPF programs to complete so that + * userspace, when we return to it, knows that all programs + * that could be running use the new map value. + */ + if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS || + map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS) + synchronize_rcu(); +} + +static int bpf_map_update_value(struct bpf_map *map, struct fd f, void *key, + void *value, __u64 flags) +{ + int err; + /* Need to create a kthread, thus must support schedule */ + if (bpf_map_is_dev_bound(map)) { + return bpf_map_offload_update_elem(map, key, value, flags); + } else if (map->map_type == BPF_MAP_TYPE_CPUMAP || + map->map_type == BPF_MAP_TYPE_SOCKHASH || + map->map_type == BPF_MAP_TYPE_SOCKMAP) { + return map->ops->map_update_elem(map, key, value, flags); + } + + /* must increment bpf_prog_active to avoid kprobe+bpf triggering from + * inside bpf map update or delete otherwise deadlocks are possible + */ + preempt_disable(); + __this_cpu_inc(bpf_prog_active); + if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || + map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { + err = bpf_percpu_hash_update(map, key, value, flags); + } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { + err = bpf_percpu_array_update(map, key, value, flags); + } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { + err = bpf_percpu_cgroup_storage_update(map, key, value, + flags); + } else if (IS_FD_ARRAY(map)) { + rcu_read_lock(); + err = bpf_fd_array_map_update_elem(map, f.file, key, value, + flags); + rcu_read_unlock(); + } else if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { + rcu_read_lock(); + err = bpf_fd_htab_map_update_elem(map, f.file, key, value, + flags); + rcu_read_unlock(); + } else if (map->map_type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) { + /* rcu_read_lock() is not needed */ + err = bpf_fd_reuseport_array_update_elem(map, key, value, + flags); + } else if (map->map_type == BPF_MAP_TYPE_QUEUE || + map->map_type == BPF_MAP_TYPE_STACK) { + err = map->ops->map_push_elem(map, value, flags); + } else { + rcu_read_lock(); + err = map->ops->map_update_elem(map, key, value, flags); + rcu_read_unlock(); + } + __this_cpu_dec(bpf_prog_active); + preempt_enable(); + maybe_wait_bpf_programs(map); + + return err; +} + +static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, + __u64 flags, bool do_delete) +{ + void *ptr; + int err; + + + if (bpf_map_is_dev_bound(map)) { + err = bpf_map_offload_lookup_elem(map, key, value); + + if (!err && do_delete) + err = bpf_map_offload_delete_elem(map, key); + + return err; + } + + preempt_disable(); + this_cpu_inc(bpf_prog_active); + if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || + map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { + err = bpf_percpu_hash_copy(map, key, value); + } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { + err = bpf_percpu_array_copy(map, key, value); + } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { + err = bpf_percpu_cgroup_storage_copy(map, key, value); + } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) { + err = bpf_stackmap_copy(map, key, value); + } else if (IS_FD_ARRAY(map)) { + err = bpf_fd_array_map_lookup_elem(map, key, value); + } else if (IS_FD_HASH(map)) { + err = bpf_fd_htab_map_lookup_elem(map, key, value); + } else if (map->map_type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) { + err = bpf_fd_reuseport_array_lookup_elem(map, key, value); + } else if (map->map_type == BPF_MAP_TYPE_QUEUE || + map->map_type == BPF_MAP_TYPE_STACK) { + err = map->ops->map_peek_elem(map, value); + } else { + rcu_read_lock(); + if (map->ops->map_lookup_elem_sys_only) + ptr = map->ops->map_lookup_elem_sys_only(map, key); + else + ptr = map->ops->map_lookup_elem(map, key); + if (IS_ERR(ptr)) { + err = PTR_ERR(ptr); + } else if (!ptr) { + err = -ENOENT; + } else { + err = 0; + if (flags & BPF_F_LOCK) + /* lock 'ptr' and copy everything but lock */ + copy_map_value_locked(map, value, ptr, true); + else + copy_map_value(map, value, ptr); + /* mask lock, since value wasn't zero inited */ + check_and_init_map_lock(map, value); + } + rcu_read_unlock(); + } + if (do_delete) + err = err ? err : map->ops->map_delete_elem(map, key); + + this_cpu_dec(bpf_prog_active); + preempt_enable(); + maybe_wait_bpf_programs(map); + + return err; +} + void *bpf_map_area_alloc(size_t size, int numa_node) { /* We really just want to fail instead of triggering OOM killer @@ -740,7 +887,7 @@ static int map_lookup_elem(union bpf_attr *attr) void __user *uvalue = u64_to_user_ptr(attr->value); int ufd = attr->map_fd; struct bpf_map *map; - void *key, *value, *ptr; + void *key, *value; u32 value_size; struct fd f; int err; @@ -772,72 +919,14 @@ static int map_lookup_elem(union bpf_attr *attr) goto err_put; } - if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY || - map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) - value_size = round_up(map->value_size, 8) * num_possible_cpus(); - else if (IS_FD_MAP(map)) - value_size = sizeof(u32); - else - value_size = map->value_size; + value_size = bpf_map_value_size(map); err = -ENOMEM; value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) goto free_key; - if (bpf_map_is_dev_bound(map)) { - err = bpf_map_offload_lookup_elem(map, key, value); - goto done; - } - - preempt_disable(); - this_cpu_inc(bpf_prog_active); - if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { - err = bpf_percpu_hash_copy(map, key, value); - } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { - err = bpf_percpu_array_copy(map, key, value); - } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { - err = bpf_percpu_cgroup_storage_copy(map, key, value); - } else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) { - err = bpf_stackmap_copy(map, key, value); - } else if (IS_FD_ARRAY(map)) { - err = bpf_fd_array_map_lookup_elem(map, key, value); - } else if (IS_FD_HASH(map)) { - err = bpf_fd_htab_map_lookup_elem(map, key, value); - } else if (map->map_type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) { - err = bpf_fd_reuseport_array_lookup_elem(map, key, value); - } else if (map->map_type == BPF_MAP_TYPE_QUEUE || - map->map_type == BPF_MAP_TYPE_STACK) { - err = map->ops->map_peek_elem(map, value); - } else { - rcu_read_lock(); - if (map->ops->map_lookup_elem_sys_only) - ptr = map->ops->map_lookup_elem_sys_only(map, key); - else - ptr = map->ops->map_lookup_elem(map, key); - if (IS_ERR(ptr)) { - err = PTR_ERR(ptr); - } else if (!ptr) { - err = -ENOENT; - } else { - err = 0; - if (attr->flags & BPF_F_LOCK) - /* lock 'ptr' and copy everything but lock */ - copy_map_value_locked(map, value, ptr, true); - else - copy_map_value(map, value, ptr); - /* mask lock, since value wasn't zero inited */ - check_and_init_map_lock(map, value); - } - rcu_read_unlock(); - } - this_cpu_dec(bpf_prog_active); - preempt_enable(); - -done: + err = bpf_map_copy_value(map, key, value, attr->flags, false); if (err) goto free_value; @@ -856,16 +945,6 @@ static int map_lookup_elem(union bpf_attr *attr) return err; } -static void maybe_wait_bpf_programs(struct bpf_map *map) -{ - /* Wait for any running BPF programs to complete so that - * userspace, when we return to it, knows that all programs - * that could be running use the new map value. - */ - if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS || - map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS) - synchronize_rcu(); -} #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags @@ -921,56 +1000,8 @@ static int map_update_elem(union bpf_attr *attr) if (copy_from_user(value, uvalue, value_size) != 0) goto free_value; - /* Need to create a kthread, thus must support schedule */ - if (bpf_map_is_dev_bound(map)) { - err = bpf_map_offload_update_elem(map, key, value, attr->flags); - goto out; - } else if (map->map_type == BPF_MAP_TYPE_CPUMAP || - map->map_type == BPF_MAP_TYPE_SOCKHASH || - map->map_type == BPF_MAP_TYPE_SOCKMAP) { - err = map->ops->map_update_elem(map, key, value, attr->flags); - goto out; - } + err = bpf_map_update_value(map, f, key, value, attr->flags); - /* must increment bpf_prog_active to avoid kprobe+bpf triggering from - * inside bpf map update or delete otherwise deadlocks are possible - */ - preempt_disable(); - __this_cpu_inc(bpf_prog_active); - if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { - err = bpf_percpu_hash_update(map, key, value, attr->flags); - } else if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { - err = bpf_percpu_array_update(map, key, value, attr->flags); - } else if (map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) { - err = bpf_percpu_cgroup_storage_update(map, key, value, - attr->flags); - } else if (IS_FD_ARRAY(map)) { - rcu_read_lock(); - err = bpf_fd_array_map_update_elem(map, f.file, key, value, - attr->flags); - rcu_read_unlock(); - } else if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) { - rcu_read_lock(); - err = bpf_fd_htab_map_update_elem(map, f.file, key, value, - attr->flags); - rcu_read_unlock(); - } else if (map->map_type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) { - /* rcu_read_lock() is not needed */ - err = bpf_fd_reuseport_array_update_elem(map, key, value, - attr->flags); - } else if (map->map_type == BPF_MAP_TYPE_QUEUE || - map->map_type == BPF_MAP_TYPE_STACK) { - err = map->ops->map_push_elem(map, value, attr->flags); - } else { - rcu_read_lock(); - err = map->ops->map_update_elem(map, key, value, attr->flags); - rcu_read_unlock(); - } - __this_cpu_dec(bpf_prog_active); - preempt_enable(); - maybe_wait_bpf_programs(map); -out: free_value: kfree(value); free_key: From patchwork Tue Nov 19 19:30:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197601 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="XtwPjozv"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXD4bkSz9sPj for ; Wed, 20 Nov 2019 06:30:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727334AbfKSTav (ORCPT ); Tue, 19 Nov 2019 14:30:51 -0500 Received: from mail-vk1-f201.google.com ([209.85.221.201]:56013 "EHLO mail-vk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727212AbfKSTat (ORCPT ); Tue, 19 Nov 2019 14:30:49 -0500 Received: by mail-vk1-f201.google.com with SMTP id n6so10250369vke.22 for ; Tue, 19 Nov 2019 11:30:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BU9DBWW2HRmi8gaM0qUEqABgtZ1XVgljZH1NSHAlVa8=; b=XtwPjozvlUmOzNljiZwCmT3QoDNXX+G5PoG1sWdDK/NV2k0FpJn6xgJibB/exhoPxN EdPpy0quITzbShznByLhs+3Ul1dlH/I8Hdw2x5zUzJ/ZmvJ8fBpmEsyjAzX/N1NlIp5K sspeqi13SrvJn5A0YnysnK4KEJidNWDHThbxZKxLdfQbCq3pyS20hxcN+3V2Btmhaphl /P1x4/Tjkv7pamotGAnYNZ/wHDLqselSXLQ7xysN7JGSi05ronR85Uj9+fZFsqDsGQkA JUBZzSjO8JjnXxIiEdTm7ldoUQTb8fc1iOKWNkuyzimr91MealwabgBn1apU9PhQKBQN /YBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BU9DBWW2HRmi8gaM0qUEqABgtZ1XVgljZH1NSHAlVa8=; b=S9HzMQfEZFC5SS2ION6yvqxv5yPg5j1zlo9DrGu2nVNBaYx0OLn8N00CcSLyTKqApu IH+YhIz4otBjD/xGpXkaDn9slrOgvGiN+BRhF3w3Etiy30RwEBih38qOyLbhZDjgmASd 6jEKmKUAedTlM2Xv/cRUd9SOo7/FTve2e9zRY3+GzKkHbiUQWF9XFZe+6yYrOEwIsp8x 5pmDWHTzppnS91FNV08WSHZp+E/rSt72JKCp025DhjLKLWQh2aea0hAaSb2vvBJl+UrZ y9QTQEZzSEVOxUqidEEBUDXaJP6yijQFXgFVCxi33v4w1Jsy/xulnSAaIRbHUY57kaAb ueig== X-Gm-Message-State: APjAAAXkCv8QB1Yc8sE3HDdmXn9r2vti/TbsP6UYzCCZUA8KvkaKKqKh ClMUrbzquAMQLxNjU2V+BB59m3a6BIjk X-Google-Smtp-Source: APXvYqyQ1nwNFQEYf1+pCLtAq5u9nv/hY74ajRfzunnSJv9aAeYBDYoBf9/AAtv3yTJJpTgG2sRflKvoQCzQ X-Received: by 2002:ab0:200c:: with SMTP id v12mr20984611uak.26.1574191848119; Tue, 19 Nov 2019 11:30:48 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:29 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-3-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 2/9] bpf: add generic support for lookup and lookup_and_delete batch ops From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This commit introduces generic support for the bpf_map_lookup_batch and bpf_map_lookup_and_delete_batch ops. This implementation can be used by almost all the bpf maps since its core implementation is relying on the existing map_get_next_key, map_lookup_elem and map_delete_elem functions. The bpf syscall subcommands introduced are: BPF_MAP_LOOKUP_BATCH BPF_MAP_LOOKUP_AND_DELETE_BATCH The UAPI attribute is: struct { /* struct used by BPF_MAP_*_BATCH commands */ __aligned_u64 in_batch; /* start batch, * NULL to start from beginning */ __aligned_u64 out_batch; /* output: next start batch */ __aligned_u64 keys; __aligned_u64 values; __u32 count; /* input/output: * input: # of key/value * elements * output: # of filled elements */ __u32 map_fd; __u64 elem_flags; __u64 flags; } batch; in_batch/out_batch are opaque values use to communicate between user/kernel space, in_batch/out_batch must be of key_size length. To start iterating from the beginning in_batch must be null, count is the # of key/value elements to retrieve. Note that the 'keys' buffer must be a buffer of key_size * count size and the 'values' buffer must be value_size * count, where value_size must be aligned to 8 bytes by userspace if it's dealing with percpu maps. 'count' will contain the number of keys/values successfully retrieved. Note that 'count' is an input/output variable and it can contain a lower value after a call. If there's no more entries to retrieve, ENOENT will be returned. If error is ENOENT, count might be > 0 in case it copied some values but there were no more entries to retrieve. Note that if the return code is an error and not -EFAULT, count indicates the number of elements successfully processed. Suggested-by: Stanislav Fomichev Signed-off-by: Brian Vazquez Signed-off-by: Yonghong Song --- include/linux/bpf.h | 11 +++ include/uapi/linux/bpf.h | 19 +++++ kernel/bpf/syscall.c | 176 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 206 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5b81cde47314e..767a823dbac74 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -41,6 +41,11 @@ struct bpf_map_ops { int (*map_get_next_key)(struct bpf_map *map, void *key, void *next_key); void (*map_release_uref)(struct bpf_map *map); void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key); + int (*map_lookup_batch)(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr); + int (*map_lookup_and_delete_batch)(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr); /* funcs callable from userspace and from eBPF programs */ void *(*map_lookup_elem)(struct bpf_map *map, void *key); @@ -797,6 +802,12 @@ void bpf_map_charge_move(struct bpf_map_memory *dst, void *bpf_map_area_alloc(size_t size, int numa_node); void bpf_map_area_free(void *base); void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); +int generic_map_lookup_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr); +int generic_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr); extern int sysctl_unprivileged_bpf_disabled; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 4842a134b202a..e60b7b7cda61a 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -107,6 +107,8 @@ enum bpf_cmd { BPF_MAP_LOOKUP_AND_DELETE_ELEM, BPF_MAP_FREEZE, BPF_BTF_GET_NEXT_ID, + BPF_MAP_LOOKUP_BATCH, + BPF_MAP_LOOKUP_AND_DELETE_BATCH, }; enum bpf_map_type { @@ -400,6 +402,23 @@ union bpf_attr { __u64 flags; }; + struct { /* struct used by BPF_MAP_*_BATCH commands */ + __aligned_u64 in_batch; /* start batch, + * NULL to start from beginning + */ + __aligned_u64 out_batch; /* output: next start batch */ + __aligned_u64 keys; + __aligned_u64 values; + __u32 count; /* input/output: + * input: # of key/value + * elements + * output: # of filled elements + */ + __u32 map_fd; + __u64 elem_flags; + __u64 flags; + } batch; + struct { /* anonymous struct used by BPF_PROG_LOAD command */ __u32 prog_type; /* one of enum bpf_prog_type */ __u32 insn_cnt; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index cc714c9d5b4cc..d0d3d0e0eaca4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1127,6 +1127,124 @@ static int map_get_next_key(union bpf_attr *attr) return err; } +static int __generic_map_lookup_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr, + bool do_delete) +{ + void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch); + void __user *uobatch = u64_to_user_ptr(attr->batch.out_batch); + void __user *values = u64_to_user_ptr(attr->batch.values); + void __user *keys = u64_to_user_ptr(attr->batch.keys); + void *buf, *prev_key, *key, *value; + u32 value_size, cp, max_count; + bool first_key = false; + int err, retry = 3; + + if (attr->batch.elem_flags & ~BPF_F_LOCK) + return -EINVAL; + + if ((attr->batch.elem_flags & BPF_F_LOCK) && + !map_value_has_spin_lock(map)) { + err = -EINVAL; + goto err_put; + } + + if (map->map_type == BPF_MAP_TYPE_QUEUE || + map->map_type == BPF_MAP_TYPE_STACK) { + err = -ENOTSUPP; + goto err_put; + } + + value_size = bpf_map_value_size(map); + + max_count = attr->batch.count; + if (!max_count) + return 0; + + err = -ENOMEM; + buf = kmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN); + if (!buf) + goto err_put; + + err = -EFAULT; + first_key = false; + if (ubatch && copy_from_user(buf, ubatch, map->key_size)) + goto free_buf; + key = buf; + value = key + map->key_size; + if (!ubatch) { + prev_key = NULL; + first_key = true; + } + + + for (cp = 0; cp < max_count; cp++) { + if (cp || first_key) { + rcu_read_lock(); + err = map->ops->map_get_next_key(map, prev_key, key); + rcu_read_unlock(); + if (err) + break; + } + err = bpf_map_copy_value(map, key, value, + attr->batch.elem_flags, do_delete); + + if (err == -ENOENT) { + if (retry) { + retry--; + continue; + } + err = -EINTR; + break; + } + + if (err) + goto free_buf; + + if (copy_to_user(keys + cp * map->key_size, key, + map->key_size)) { + err = -EFAULT; + goto free_buf; + } + if (copy_to_user(values + cp * value_size, value, value_size)) { + err = -EFAULT; + goto free_buf; + } + + prev_key = key; + retry = 3; + } + if (!err) { + rcu_read_lock(); + err = map->ops->map_get_next_key(map, prev_key, key); + rcu_read_unlock(); + } + + if ((copy_to_user(&uattr->batch.count, &cp, sizeof(cp)) || + (copy_to_user(uobatch, key, map->key_size)))) + err = -EFAULT; + +free_buf: + kfree(buf); +err_put: + return err; +} + +int generic_map_lookup_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __generic_map_lookup_batch(map, attr, uattr, false); +} + +int generic_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __generic_map_lookup_batch(map, attr, uattr, true); +} + #define BPF_MAP_LOOKUP_AND_DELETE_ELEM_LAST_FIELD value static int map_lookup_and_delete_elem(union bpf_attr *attr) @@ -2956,6 +3074,57 @@ static int bpf_task_fd_query(const union bpf_attr *attr, return err; } +#define BPF_MAP_BATCH_LAST_FIELD batch.flags + +#define BPF_DO_BATCH(fn) \ + do { \ + if (!fn) { \ + err = -ENOTSUPP; \ + goto err_put; \ + } \ + err = fn(map, attr, uattr); \ + } while (0) + +static int bpf_map_do_batch(const union bpf_attr *attr, + union bpf_attr __user *uattr, + int cmd) +{ + struct bpf_map *map; + int err, ufd; + struct fd f; + + if (CHECK_ATTR(BPF_MAP_BATCH)) + return -EINVAL; + + ufd = attr->batch.map_fd; + f = fdget(ufd); + map = __bpf_map_get(f); + if (IS_ERR(map)) + return PTR_ERR(map); + + if ((cmd == BPF_MAP_LOOKUP_BATCH || + cmd == BPF_MAP_LOOKUP_AND_DELETE_BATCH) && + !(map_get_sys_perms(map, f) & FMODE_CAN_READ)) { + err = -EPERM; + goto err_put; + } + + if (cmd != BPF_MAP_LOOKUP_BATCH && + !(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) { + err = -EPERM; + goto err_put; + } + + if (cmd == BPF_MAP_LOOKUP_BATCH) + BPF_DO_BATCH(map->ops->map_lookup_batch); + else + BPF_DO_BATCH(map->ops->map_lookup_and_delete_batch); + +err_put: + fdput(f); + return err; +} + SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size) { union bpf_attr attr = {}; @@ -3053,6 +3222,13 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz case BPF_MAP_LOOKUP_AND_DELETE_ELEM: err = map_lookup_and_delete_elem(&attr); break; + case BPF_MAP_LOOKUP_BATCH: + err = bpf_map_do_batch(&attr, uattr, BPF_MAP_LOOKUP_BATCH); + break; + case BPF_MAP_LOOKUP_AND_DELETE_BATCH: + err = bpf_map_do_batch(&attr, uattr, + BPF_MAP_LOOKUP_AND_DELETE_BATCH); + break; default: err = -EINVAL; break; From patchwork Tue Nov 19 19:30:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197615 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="jcFKJmRc"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbYB4FzSz9sPj for ; Wed, 20 Nov 2019 06:31:42 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727363AbfKSTax (ORCPT ); Tue, 19 Nov 2019 14:30:53 -0500 Received: from mail-pf1-f201.google.com ([209.85.210.201]:44656 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727212AbfKSTav (ORCPT ); Tue, 19 Nov 2019 14:30:51 -0500 Received: by mail-pf1-f201.google.com with SMTP id 2so17506342pfx.11 for ; Tue, 19 Nov 2019 11:30:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=paPq1QV4mwCwfbyX9rPU2ygE8/29H1pmhKP3AAocJPk=; b=jcFKJmRckPuxfkTZewDK1THPa2GajsqYtYyYWJUFvN+x2hxPQ0T0R0+4MHha838m8L 2YwAo3N/DZB745c5cEgwnhzJB7ThpRjc4HwV65wCKKJNchX7TwUbNNEyhaIgyonCzNVw A8WfzOE72FoPmz/rOw9kr6MmWyGjtdGWA2ZHhwyC4Mz5TG+Kl1C5vtruoY7ZL8P8fhih 9IUNU8aj4qN8ZVvIRpYklYFceRvBBvYbRgHmu75EMxH5z1nZfPnkThKORQJfFvzMXfa0 cuf/HNw38shj7eaNIPdtWNHbQ4y4PKaARk+EkZ3uxziLvUydE9DuZz/wsmL0/X3CLtw0 uiyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=paPq1QV4mwCwfbyX9rPU2ygE8/29H1pmhKP3AAocJPk=; b=BQAr0XtcposEuEiHHHdeerhGHC8bQCVrEKYwMP6+bgBcuoeFngxx7T9TUvWXi8Nly+ MjOirBDSGVBYaNXQowyxXZwjy6aDgP28ia5zRDlyo1f7BWO1+4kDHYjiPDcCPAFCQcWV IQffNX43iAxqmnTBN79c8JxFAvI6xCFEgK/PqpHP03QqzEbgN0Y6PHqTDN661J/S9o6s DPVbUF4YRsdqNhFn3aMkPaDT9ARdDjh+MiD3wnueJ53+7pxM7PgcMRMXIC+WOU9Snp7u 29vfw5ol+ElFeBb80fCzUx/Bz0jzHzr+JKPgWYJ3rhfeL6o730rrbg6zUBdrmwunglup lW8A== X-Gm-Message-State: APjAAAVjw1lm7HFIB7bFLa3c1tYAfTNGszMz/LGq5cDHR7Aakdv7Tbbf StXIDXVt5dcKRqJV7qPeQK4DZsVigepe X-Google-Smtp-Source: APXvYqxqqYhaHewEXYbDwlpghe66juXiw0KNBbLxkJfqK0Ye8IrnsM0CIba2uNVuLeoq17Wia4hv68S7T+2P X-Received: by 2002:a63:545f:: with SMTP id e31mr3901108pgm.236.1574191850913; Tue, 19 Nov 2019 11:30:50 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:30 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-4-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 3/9] bpf: add generic support for update and delete batch ops From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit adds generic support for update and delete batch ops that can be used for almost all the bpf maps. These commands share the same UAPI attr that lookup and lookup_and_delete batch ops use and the syscall commands are: BPF_MAP_UPDATE_BATCH BPF_MAP_DELETE_BATCH The main difference between update/delete and lookup/lookup_and_delete batch ops is that for update/delete keys/values must be specified for userspace and because of that, neither in_batch nor out_batch are used. Suggested-by: Stanislav Fomichev Signed-off-by: Brian Vazquez Signed-off-by: Yonghong Song --- include/linux/bpf.h | 10 ++++ include/uapi/linux/bpf.h | 2 + kernel/bpf/syscall.c | 126 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 137 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 767a823dbac74..96a19e1fd2b5b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -46,6 +46,10 @@ struct bpf_map_ops { int (*map_lookup_and_delete_batch)(struct bpf_map *map, const union bpf_attr *attr, union bpf_attr __user *uattr); + int (*map_update_batch)(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr); + int (*map_delete_batch)(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr); /* funcs callable from userspace and from eBPF programs */ void *(*map_lookup_elem)(struct bpf_map *map, void *key); @@ -808,6 +812,12 @@ int generic_map_lookup_batch(struct bpf_map *map, int generic_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr, union bpf_attr __user *uattr); +int generic_map_update_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr); +int generic_map_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr); extern int sysctl_unprivileged_bpf_disabled; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index e60b7b7cda61a..0f6ff0c4d79dd 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -109,6 +109,8 @@ enum bpf_cmd { BPF_BTF_GET_NEXT_ID, BPF_MAP_LOOKUP_BATCH, BPF_MAP_LOOKUP_AND_DELETE_BATCH, + BPF_MAP_UPDATE_BATCH, + BPF_MAP_DELETE_BATCH, }; enum bpf_map_type { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index d0d3d0e0eaca4..06e1bcf40fb8d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1127,6 +1127,120 @@ static int map_get_next_key(union bpf_attr *attr) return err; } +int generic_map_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + void __user *keys = u64_to_user_ptr(attr->batch.keys); + int ufd = attr->map_fd; + u32 cp, max_count; + struct fd f; + void *key; + int err; + + f = fdget(ufd); + if (attr->batch.elem_flags & ~BPF_F_LOCK) + return -EINVAL; + + if ((attr->batch.elem_flags & BPF_F_LOCK) && + !map_value_has_spin_lock(map)) { + err = -EINVAL; + goto err_put; + } + + max_count = attr->batch.count; + if (!max_count) + return 0; + + err = -ENOMEM; + for (cp = 0; cp < max_count; cp++) { + key = __bpf_copy_key(keys + cp * map->key_size, map->key_size); + if (IS_ERR(key)) { + err = PTR_ERR(key); + break; + } + + if (err) + break; + if (bpf_map_is_dev_bound(map)) { + err = bpf_map_offload_delete_elem(map, key); + break; + } + + preempt_disable(); + __this_cpu_inc(bpf_prog_active); + rcu_read_lock(); + err = map->ops->map_delete_elem(map, key); + rcu_read_unlock(); + __this_cpu_dec(bpf_prog_active); + preempt_enable(); + maybe_wait_bpf_programs(map); + if (err) + break; + } + if (copy_to_user(&uattr->batch.count, &cp, sizeof(cp))) + err = -EFAULT; +err_put: + return err; +} +int generic_map_update_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + void __user *values = u64_to_user_ptr(attr->batch.values); + void __user *keys = u64_to_user_ptr(attr->batch.keys); + u32 value_size, cp, max_count; + int ufd = attr->map_fd; + void *key, *value; + struct fd f; + int err; + + f = fdget(ufd); + if (attr->batch.elem_flags & ~BPF_F_LOCK) + return -EINVAL; + + if ((attr->batch.elem_flags & BPF_F_LOCK) && + !map_value_has_spin_lock(map)) { + err = -EINVAL; + goto err_put; + } + + value_size = bpf_map_value_size(map); + + max_count = attr->batch.count; + if (!max_count) + return 0; + + err = -ENOMEM; + value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); + if (!value) + goto err_put; + + for (cp = 0; cp < max_count; cp++) { + key = __bpf_copy_key(keys + cp * map->key_size, map->key_size); + if (IS_ERR(key)) { + err = PTR_ERR(key); + break; + } + err = -EFAULT; + if (copy_from_user(value, values + cp * value_size, value_size)) + break; + + err = bpf_map_update_value(map, f, key, value, + attr->batch.elem_flags); + + if (err) + break; + } + + if (copy_to_user(&uattr->batch.count, &cp, sizeof(cp))) + err = -EFAULT; + + kfree(value); +err_put: + return err; +} + static int __generic_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, union bpf_attr __user *uattr, @@ -3117,8 +3231,12 @@ static int bpf_map_do_batch(const union bpf_attr *attr, if (cmd == BPF_MAP_LOOKUP_BATCH) BPF_DO_BATCH(map->ops->map_lookup_batch); - else + else if (cmd == BPF_MAP_LOOKUP_AND_DELETE_BATCH) BPF_DO_BATCH(map->ops->map_lookup_and_delete_batch); + else if (cmd == BPF_MAP_UPDATE_BATCH) + BPF_DO_BATCH(map->ops->map_update_batch); + else + BPF_DO_BATCH(map->ops->map_delete_batch); err_put: fdput(f); @@ -3229,6 +3347,12 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz err = bpf_map_do_batch(&attr, uattr, BPF_MAP_LOOKUP_AND_DELETE_BATCH); break; + case BPF_MAP_UPDATE_BATCH: + err = bpf_map_do_batch(&attr, uattr, BPF_MAP_UPDATE_BATCH); + break; + case BPF_MAP_DELETE_BATCH: + err = bpf_map_do_batch(&attr, uattr, BPF_MAP_DELETE_BATCH); + break; default: err = -EINVAL; break; From patchwork Tue Nov 19 19:30:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197613 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="KKfy6cGF"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXv6nqCz9sR4 for ; Wed, 20 Nov 2019 06:31:27 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727467AbfKSTa4 (ORCPT ); Tue, 19 Nov 2019 14:30:56 -0500 Received: from mail-pg1-f202.google.com ([209.85.215.202]:51855 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbfKSTaz (ORCPT ); Tue, 19 Nov 2019 14:30:55 -0500 Received: by mail-pg1-f202.google.com with SMTP id f18so4025905pgh.18 for ; Tue, 19 Nov 2019 11:30:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+X4EfDiauElJOC7RvUHMTNrf0yIKJ6Yo/r1JySL4OmE=; b=KKfy6cGFJzrvbZ+zaVtoIK6lbKkij8Ez+QP+UrABS7jTH6fWRYIOLXrUNxMuEhYho2 gIJ0Z3z9EY5uAfnR/RT2mxFOPeuEJXtf0gOs2iwNjq7C89kcDwoPU+pkr6Fv7S2kIllI cgnJ4vZ95Lh6qUuqJ/b9vUDGOMJMIfF8TbS9XjBvp0KfxklTpcbFSSHiC/EoGqm4VvGT xheYmmAUWYh+0a+EgLJGRRBp61JZWwI/gvzpV9x8oFIiT98N8UwQNBptCNGx50kVH004 1+eZVNTge9RNKalRpnP80Z9HZjoDuuoslzDpFkhj0dZ0CTDwUuoGXV0HxvGcGYxjt6Rv G3Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+X4EfDiauElJOC7RvUHMTNrf0yIKJ6Yo/r1JySL4OmE=; b=jIZHZBV27VP5Qp8oA9wLsoqshM/b66Chfg8U62SW3KicVdkD9ltGyIJQHNyfrdDfot gJ6qBO2sv6AsfkajQG2wXucqKqDTLqSG3vVuIee0+9M21KCe0ACbYHXr3SXjmcwmJDqw 20dC0ofW9MMztZS5NCGXSIoaEA1X7PE8rlOg6+0wOexaP32+qBN/3+SDAcTSzI6eNBzK KKbKdRE1cO0zOnWq3nOZfwNP+7Oxad9scPpb+Fh5u749DuZO5SRma11hEYKQY551hdW3 wL9Hi/ClskEZ4BdVibUzl7jdbaC4cQ+1onq8EBasWYjHc8aBd+NKHYW/U7v9qrOTJnYM yRfg== X-Gm-Message-State: APjAAAXExxXTXOURdLDlzl70jqWyYK+Z6nRjujqXnxQMhkdbuZduL73T Q5Jz7H7KtUzIthxJ0t0upfZjN7vrYLbW X-Google-Smtp-Source: APXvYqzXkr5QL8TzNPUeAu0p3xccRyiDCHLjKy9+fE96m5hofERZicATy3cDVLslLNxJKgSdiDuS1g9cr/Kq X-Received: by 2002:a63:3587:: with SMTP id c129mr7516833pga.211.1574191853520; Tue, 19 Nov 2019 11:30:53 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:31 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-5-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 4/9] bpf: add lookup and updated batch ops to arraymap From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This adds the generic batch ops functionality to bpf arraymap, note that since deletion is not a valid operation for arraymap, only batch and lookup are added. Signed-off-by: Brian Vazquez --- kernel/bpf/arraymap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 1c65ce0098a95..680d4e99ef583 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -457,6 +457,8 @@ const struct bpf_map_ops array_map_ops = { .map_direct_value_meta = array_map_direct_value_meta, .map_seq_show_elem = array_map_seq_show_elem, .map_check_btf = array_map_check_btf, + .map_lookup_batch = generic_map_lookup_batch, + .map_update_batch = generic_map_update_batch, }; const struct bpf_map_ops percpu_array_map_ops = { From patchwork Tue Nov 19 19:30:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197603 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="nI0IQZFS"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXM5b2tz9sPv for ; Wed, 20 Nov 2019 06:30:59 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727324AbfKSTa6 (ORCPT ); Tue, 19 Nov 2019 14:30:58 -0500 Received: from mail-pf1-f201.google.com ([209.85.210.201]:50780 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727450AbfKSTa5 (ORCPT ); Tue, 19 Nov 2019 14:30:57 -0500 Received: by mail-pf1-f201.google.com with SMTP id e13so17493973pff.17 for ; Tue, 19 Nov 2019 11:30:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hBTV2OVOS+v8kQKNTas6sjat//bLzlmUxL3n7cHoG5Y=; b=nI0IQZFSv4DaaqUXa2pdOg8ruFkFRjPfPzFg/21BQnYhJX90E5O8wd1AMdIJgBHtd1 dTbMjbCEGvxdSor2owRB/cf1xHVImFpw8q3U1jjzMLg002nB4uX4VLq4q5qzdd0RMaXU v41bgl/klHFQWORHO3tuNB7XcyyBu6sAYucSo1T7HutZ6wmUbb5XCd2WQzYgX74EoTGz 3WmP+y9+NCcflnyaDyAYDfMnpeSn6+d9kobjZWq0q04r7n4xd6hv3h0chkH4kzoO9v7d v3hDK6g42k1MvYGFwtAJD7OrkMxhZSAAjyU6XaDZmAbSXQicDoI4fh+EF2LMbXJxj0bQ e/Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hBTV2OVOS+v8kQKNTas6sjat//bLzlmUxL3n7cHoG5Y=; b=tFWqGUtPrveNdXRhejofVj4ZIAkQfffmub0PFQJbxibco9mtR6MHf7uwwZWDsaz+1d IUvHjBzoscxLh5gt6VR18dkP6DZuDB/rizEcnruoGawVBTYuJcy38ix3Z2NRwSM+iEbI JiwS/Im/rNafbUYFp3qhRWnI4WpBwpnrVYO6FiwNgyM/o8p+Y2iiDA0dphB4cKcsAm7A GhIp2WHhCoIHaCybjDFjjqCZsLn7aPgyF4qg94tAZiDNnOPe7woM9YCLlZUz3mW8QOi/ Sh50G3elwPhQYzMI+ZdbSe3ZmH3Bjs7zn69Ckj3Mrr6V7rj2g+ap1iIheL79FHjc0Zqy 8b1w== X-Gm-Message-State: APjAAAWzGyBPDbeBJsdXgEhaNxZcq+5dIaoMkOWEutq5Na0L6k/BUOY9 ORv70/uhpHWGbW3PvQ6ACekLdI+4nJ0c X-Google-Smtp-Source: APXvYqxEQoUq0tZ0vbSpqMdAFxze3RLmzT24K6u4LV5ZCaazObFjteIckYTlo/6On3IkwBmunX09iwvZA9wZ X-Received: by 2002:a63:4562:: with SMTP id u34mr7239596pgk.399.1574191855970; Tue, 19 Nov 2019 11:30:55 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:32 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-6-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 5/9] bpf: add batch ops to all htab bpf map From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yonghong Song htab can't use generic batch support due some problematic behaviours inherent to the data structre, i.e. while iterating the bpf map a concurrent program might delete the next entry that batch was about to use, in that case there's no easy solution to retrieve the next entry, the issue has been discussed multiple times (see [1] and [2]). The only way hmap can be traversed without the problem previously exposed is by making sure that the map is traversing entire buckets. This commit implements those strict requirements for hmap, the implementation follows the same interaction that generic support with some exceptions: - If keys/values buffer are not big enough to traverse a bucket, ENOSPC will be returned. - out_batch contains the value of the next bucket in the iteration, not the next key, but this is transparent for the user since the user should never use out_batch for other than bpf batch syscalls. Note that only lookup and lookup_and_delete batch ops require the hmap specific implementation, update/delete batch ops can be the generic ones. [1] https://lore.kernel.org/bpf/20190724165803.87470-1-brianvv@google.com/ [2] https://lore.kernel.org/bpf/20190906225434.3635421-1-yhs@fb.com/ Signed-off-by: Yonghong Song Signed-off-by: Brian Vazquez --- kernel/bpf/hashtab.c | 244 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 244 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 22066a62c8c97..3402174b292ea 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -17,6 +17,17 @@ (BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU | BPF_F_NUMA_NODE | \ BPF_F_ACCESS_MASK | BPF_F_ZERO_SEED) +#define BATCH_OPS(_name) \ + .map_lookup_batch = \ + _name##_map_lookup_batch, \ + .map_lookup_and_delete_batch = \ + _name##_map_lookup_and_delete_batch, \ + .map_update_batch = \ + generic_map_update_batch, \ + .map_delete_batch = \ + generic_map_delete_batch + + struct bucket { struct hlist_nulls_head head; raw_spinlock_t lock; @@ -1232,6 +1243,235 @@ static void htab_map_seq_show_elem(struct bpf_map *map, void *key, rcu_read_unlock(); } +static int +__htab_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr, + bool do_delete, bool is_lru_map, + bool is_percpu) +{ + struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + u32 bucket_cnt, total, key_size, value_size, roundup_key_size; + void *keys = NULL, *values = NULL, *value, *dst_key, *dst_val; + void __user *ukeys, *uvalues, *ubatch; + u64 elem_map_flags, map_flags; + struct hlist_nulls_head *head; + struct hlist_nulls_node *n; + u32 batch, max_count, size; + unsigned long flags; + struct htab_elem *l; + struct bucket *b; + int ret = 0; + + max_count = attr->batch.count; + if (!max_count) + return 0; + + elem_map_flags = attr->batch.elem_flags; + if ((elem_map_flags & ~BPF_F_LOCK) || + ((elem_map_flags & BPF_F_LOCK) && !map_value_has_spin_lock(map))) + return -EINVAL; + + map_flags = attr->batch.flags; + if (map_flags) + return -EINVAL; + + batch = 0; + ubatch = u64_to_user_ptr(attr->batch.in_batch); + if (ubatch && copy_from_user(&batch, ubatch, sizeof(batch))) + return -EFAULT; + + if (batch >= htab->n_buckets) + return -ENOENT; + + /* We cannot do copy_from_user or copy_to_user inside + * the rcu_read_lock. Allocate enough space here. + */ + key_size = htab->map.key_size; + roundup_key_size = round_up(htab->map.key_size, 8); + value_size = htab->map.value_size; + size = round_up(value_size, 8); + if (is_percpu) + value_size = size * num_possible_cpus(); + keys = kvmalloc(key_size * max_count, GFP_USER | __GFP_NOWARN); + values = kvmalloc(value_size * max_count, GFP_USER | __GFP_NOWARN); + if (!keys || !values) { + ret = -ENOMEM; + goto out; + } + + dst_key = keys; + dst_val = values; + total = 0; + + preempt_disable(); + this_cpu_inc(bpf_prog_active); + rcu_read_lock(); + +again: + b = &htab->buckets[batch]; + head = &b->head; + raw_spin_lock_irqsave(&b->lock, flags); + + bucket_cnt = 0; + hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) + bucket_cnt++; + + if (bucket_cnt > (max_count - total)) { + if (total == 0) + ret = -ENOSPC; + goto after_loop; + } + + hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) { + memcpy(dst_key, l->key, key_size); + + if (is_percpu) { + int off = 0, cpu; + void __percpu *pptr; + + pptr = htab_elem_get_ptr(l, map->key_size); + for_each_possible_cpu(cpu) { + bpf_long_memcpy(dst_val + off, + per_cpu_ptr(pptr, cpu), size); + off += size; + } + } else { + value = l->key + roundup_key_size; + if (elem_map_flags & BPF_F_LOCK) + copy_map_value_locked(map, dst_val, value, + true); + else + copy_map_value(map, dst_val, value); + check_and_init_map_lock(map, dst_val); + } + dst_key += key_size; + dst_val += value_size; + total++; + } + + if (do_delete) { + hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) { + hlist_nulls_del_rcu(&l->hash_node); + if (is_lru_map) + bpf_lru_push_free(&htab->lru, &l->lru_node); + else + free_htab_elem(htab, l); + } + } + + batch++; + if (batch >= htab->n_buckets) { + ret = -ENOENT; + goto after_loop; + } + + raw_spin_unlock_irqrestore(&b->lock, flags); + goto again; + +after_loop: + raw_spin_unlock_irqrestore(&b->lock, flags); + + rcu_read_unlock(); + this_cpu_dec(bpf_prog_active); + preempt_enable(); + + if (ret && ret != -ENOENT) + goto out; + + /* copy data back to user */ + ukeys = u64_to_user_ptr(attr->batch.keys); + uvalues = u64_to_user_ptr(attr->batch.values); + ubatch = u64_to_user_ptr(attr->batch.out_batch); + if (copy_to_user(ubatch, &batch, sizeof(batch)) || + copy_to_user(ukeys, keys, total * key_size) || + copy_to_user(uvalues, values, total * value_size) || + put_user(total, &uattr->batch.count)) + ret = -EFAULT; + +out: + kvfree(keys); + kvfree(values); + return ret; +} + +static int +htab_percpu_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, false, + false, true); +} + +static int +htab_percpu_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, true, + false, true); +} + +static int +htab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, false, + false, false); +} + +static int +htab_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, true, + false, false); +} + +static int +htab_map_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return generic_map_delete_batch(map, attr, uattr); +} + +static int +htab_lru_percpu_map_lookup_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, false, + true, true); +} + +static int +htab_lru_percpu_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, true, + true, true); +} + +static int +htab_lru_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, false, + true, false); +} + +static int +htab_lru_map_lookup_and_delete_batch(struct bpf_map *map, + const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return __htab_map_lookup_and_delete_batch(map, attr, uattr, true, + true, false); +} + const struct bpf_map_ops htab_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, @@ -1242,6 +1482,7 @@ const struct bpf_map_ops htab_map_ops = { .map_delete_elem = htab_map_delete_elem, .map_gen_lookup = htab_map_gen_lookup, .map_seq_show_elem = htab_map_seq_show_elem, + BATCH_OPS(htab), }; const struct bpf_map_ops htab_lru_map_ops = { @@ -1255,6 +1496,7 @@ const struct bpf_map_ops htab_lru_map_ops = { .map_delete_elem = htab_lru_map_delete_elem, .map_gen_lookup = htab_lru_map_gen_lookup, .map_seq_show_elem = htab_map_seq_show_elem, + BATCH_OPS(htab_lru), }; /* Called from eBPF program */ @@ -1368,6 +1610,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { .map_update_elem = htab_percpu_map_update_elem, .map_delete_elem = htab_map_delete_elem, .map_seq_show_elem = htab_percpu_map_seq_show_elem, + BATCH_OPS(htab_percpu), }; const struct bpf_map_ops htab_lru_percpu_map_ops = { @@ -1379,6 +1622,7 @@ const struct bpf_map_ops htab_lru_percpu_map_ops = { .map_update_elem = htab_lru_percpu_map_update_elem, .map_delete_elem = htab_lru_map_delete_elem, .map_seq_show_elem = htab_percpu_map_seq_show_elem, + BATCH_OPS(htab_lru_percpu), }; static int fd_htab_map_alloc_check(union bpf_attr *attr) From patchwork Tue Nov 19 19:30:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197604 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="qRnAj9sL"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXR3dyFz9sPv for ; Wed, 20 Nov 2019 06:31:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727522AbfKSTbC (ORCPT ); Tue, 19 Nov 2019 14:31:02 -0500 Received: from mail-pf1-f202.google.com ([209.85.210.202]:39409 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727498AbfKSTbB (ORCPT ); Tue, 19 Nov 2019 14:31:01 -0500 Received: by mail-pf1-f202.google.com with SMTP id z2so16705831pfg.6 for ; Tue, 19 Nov 2019 11:30:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jXrWlSvxSIV7SLutAc92WU1LozEt5/o5gmsENSdeMlw=; b=qRnAj9sLFjfyidZkY77ZuLNvqnfOM2c7OsFHowTq1B61urCpWGgj83PZqUwv5U68j9 nmbEI+CVIVaMsYYe7ObuAZuMsvdEgJcWUcmUpWqODQKgoVp4beMP4FHCd/GVTG5XMZ/f Dg8ad+YPlG7acNGLIInEzKl2qcrLhEYv0aIIWJ7hUcnXBaGP2yBNDg9GRZc4eNNhWMmB nbvi4sxTJTtMIcGb+pnb7UEcsm5lMb6mPQB1eShne6yUnHElJDYs2tPrnrSnvEtZlRCj dt1iusYpASeJ8MG0scyYp1twH2lT10O1Xa13zwoRQ1RBfonUk3C0quB6b2pe1QhIDQwI diEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jXrWlSvxSIV7SLutAc92WU1LozEt5/o5gmsENSdeMlw=; b=Ng/nVAv7C5V6bqn3udcYayZ7jCoUPxozZe27PsRkFeSjBW0wbNzCB7TXfytmD7xvPm rqYWXCdWRI2tmJchcPvnHE0BoBgeBE6IVQgKJsmfLOQ0EEHy2iCsyVU3DGtmq9frw3mB 2BBzBHC2FOsyNGf1xVq+x8aRGR2t9Rj7pqpktmE4QzuQ7XRwgbar/uM7zybihc7X9mOx ERWo8dJdc67V4sB6J1YVW0/7Z5bCEaxiwZhPI373OgPVc/XRvMrKFHSHodYje0tEDvSv TxmfAEtrg3x84mJ2nV0VHZvk0aovGBALY04K0FtzE9SmWCFuSWNzBQaB5NWM7Iki0Jiv Lu/g== X-Gm-Message-State: APjAAAXvD4FxulPyd2KMIoJpsJU5a7VtI6eIXjBfHn03AMIIkIJlaRxN KGnWAzgyqmKBrsgUJzieKkZYVFkiSdlI X-Google-Smtp-Source: APXvYqwL5tKN6MLbbinOhEr3EbjZs9T1WyD1ZfcTtUFtgeNZqCEfxlfny6rPELtrE4bo+lnFpV2eVPVoR/k8 X-Received: by 2002:a63:ff66:: with SMTP id s38mr7329802pgk.84.1574191858436; Tue, 19 Nov 2019 11:30:58 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:33 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-7-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 6/9] tools/bpf: sync uapi header bpf.h From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yonghong Song sync uapi header include/uapi/linux/bpf.h to tools/include/uapi/linux/bpf.h Signed-off-by: Yonghong Song --- tools/include/uapi/linux/bpf.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 4842a134b202a..0f6ff0c4d79dd 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -107,6 +107,10 @@ enum bpf_cmd { BPF_MAP_LOOKUP_AND_DELETE_ELEM, BPF_MAP_FREEZE, BPF_BTF_GET_NEXT_ID, + BPF_MAP_LOOKUP_BATCH, + BPF_MAP_LOOKUP_AND_DELETE_BATCH, + BPF_MAP_UPDATE_BATCH, + BPF_MAP_DELETE_BATCH, }; enum bpf_map_type { @@ -400,6 +404,23 @@ union bpf_attr { __u64 flags; }; + struct { /* struct used by BPF_MAP_*_BATCH commands */ + __aligned_u64 in_batch; /* start batch, + * NULL to start from beginning + */ + __aligned_u64 out_batch; /* output: next start batch */ + __aligned_u64 keys; + __aligned_u64 values; + __u32 count; /* input/output: + * input: # of key/value + * elements + * output: # of filled elements + */ + __u32 map_fd; + __u64 elem_flags; + __u64 flags; + } batch; + struct { /* anonymous struct used by BPF_PROG_LOAD command */ __u32 prog_type; /* one of enum bpf_prog_type */ __u32 insn_cnt; From patchwork Tue Nov 19 19:30:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197609 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="NM4XR6WW"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXm211Pz9sPj for ; Wed, 20 Nov 2019 06:31:20 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727544AbfKSTbE (ORCPT ); Tue, 19 Nov 2019 14:31:04 -0500 Received: from mail-qt1-f201.google.com ([209.85.160.201]:41383 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727515AbfKSTbC (ORCPT ); Tue, 19 Nov 2019 14:31:02 -0500 Received: by mail-qt1-f201.google.com with SMTP id 2so15322642qtg.8 for ; Tue, 19 Nov 2019 11:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+lN+OnV/OGJwrxgrmDkzu2oxVw2ucT8UYsYGvvebriE=; b=NM4XR6WWNRFmctuup2sP4jTFMScoszHL0nwBVfSPrvc1hbAKi1GXF75Ngf634ArcMF llrHQtDYx0UjImSHtxD2z9RK5M1KdBOeTVIsUWTIJZXbSmvem3sEHxWz27I1vig7o6ea VGrW2X6jeyynS7LkI/0IcRJpnIfRUR4qZujUzTbdeLb89zD+6oS12LzXAzEpLIILR9fo fBWk7Uc1JfRL5AB5szAXP4ag4PvOx7nIIniJVo83+U4X9Pvu9tfMRx58vTGi1A6wFJC6 a9JFAzmx8xpL+V+rgG6cNcLTHH/03nuozbgzj1cmsV7swyn/p39g2aO7cSkrinAK3ope Y4Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+lN+OnV/OGJwrxgrmDkzu2oxVw2ucT8UYsYGvvebriE=; b=A/SoiP8C+tRLQigO2h147W0f9zylm9950IReNEQHpPBrwMZNjwVNZdlEcOmD064NAd WvMxJ4hGDYwM9ModOLhof1vj4Su78nw6k6WM4srtDQ7UDbpLV7Z84cCtSmCwIDgre3KQ viOn8AdE3JQSIFGKF1t6JcwkWa0pUSCLfnDxSNPM79JzVUOV0tSH4jlwC6H09DxeEmZh ns6XNVEYgpHjYL9kaxI8Qq/V3GnK8XSJ0B/nXDUOK0JnkGamD74rv8OmxL6ginekglY4 u2Ol+D+pqI8ayiOuSZFEN23TqZze/AwAkncgFFlJh3aYln6aXR++opHf9l/wjmNV/9Up 9V/g== X-Gm-Message-State: APjAAAWzn/zhQBjUXEOxM3rLoIYn1B+kS27ll7JPLrB1JJ7nUmYo10BZ DE07DBpFyaWbg2iSQbVonVSYg2UuEaec X-Google-Smtp-Source: APXvYqxf+GNA8hu6R+vVbmfguwH7wdM89U2nb3UrGK4fF8N3LhjgyQzIbjEIuhA5rVO3LUKs+aI2rRRkLbzy X-Received: by 2002:ac8:783:: with SMTP id l3mr34181632qth.257.1574191861009; Tue, 19 Nov 2019 11:31:01 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:34 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-8-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 7/9] libbpf: add libbpf support to batch ops From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Yonghong Song Added four libbpf API functions to support map batch operations: . int bpf_map_delete_batch( ... ) . int bpf_map_lookup_batch( ... ) . int bpf_map_lookup_and_delete_batch( ... ) . int bpf_map_update_batch( ... ) Signed-off-by: Yonghong Song --- tools/lib/bpf/bpf.c | 61 ++++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/bpf.h | 14 +++++++++ tools/lib/bpf/libbpf.map | 4 +++ 3 files changed, 79 insertions(+) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 98596e15390fb..9acd9309b47b3 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -443,6 +443,67 @@ int bpf_map_freeze(int fd) return sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr)); } +static int bpf_map_batch_common(int cmd, int fd, void *in_batch, + void *out_batch, void *keys, void *values, + __u32 *count, __u64 elem_flags, + __u64 flags) +{ + union bpf_attr attr = {}; + int ret; + + memset(&attr, 0, sizeof(attr)); + attr.batch.map_fd = fd; + attr.batch.in_batch = ptr_to_u64(in_batch); + attr.batch.out_batch = ptr_to_u64(out_batch); + attr.batch.keys = ptr_to_u64(keys); + attr.batch.values = ptr_to_u64(values); + if (count) + attr.batch.count = *count; + attr.batch.elem_flags = elem_flags; + attr.batch.flags = flags; + + ret = sys_bpf(cmd, &attr, sizeof(attr)); + if (count) + *count = attr.batch.count; + + return ret; +} + +int bpf_map_delete_batch(int fd, void *in_batch, void *out_batch, __u32 *count, + __u64 elem_flags, __u64 flags) +{ + return bpf_map_batch_common(BPF_MAP_DELETE_BATCH, fd, in_batch, + out_batch, NULL, NULL, count, + elem_flags, flags); +} + +int bpf_map_lookup_batch(int fd, void *in_batch, void *out_batch, void *keys, + void *values, __u32 *count, + __u64 elem_flags, __u64 flags) +{ + return bpf_map_batch_common(BPF_MAP_LOOKUP_BATCH, fd, in_batch, + out_batch, keys, values, count, + elem_flags, flags); +} + +int bpf_map_lookup_and_delete_batch(int fd, void *in_batch, void *out_batch, + void *keys, void *values, + __u32 *count, __u64 elem_flags, + __u64 flags) +{ + return bpf_map_batch_common(BPF_MAP_LOOKUP_AND_DELETE_BATCH, + fd, in_batch, out_batch, keys, values, + count, elem_flags, flags); +} + +int bpf_map_update_batch(int fd, void *keys, void *values, __u32 *count, + __u64 elem_flags, __u64 flags) +{ + return bpf_map_batch_common(BPF_MAP_UPDATE_BATCH, + fd, NULL, NULL, keys, values, + count, elem_flags, flags); +} + int bpf_obj_pin(int fd, const char *pathname) { union bpf_attr attr; diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 3c791fa8e68e8..3ec63384400f1 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -126,6 +126,20 @@ LIBBPF_API int bpf_map_lookup_and_delete_elem(int fd, const void *key, LIBBPF_API int bpf_map_delete_elem(int fd, const void *key); LIBBPF_API int bpf_map_get_next_key(int fd, const void *key, void *next_key); LIBBPF_API int bpf_map_freeze(int fd); +LIBBPF_API int bpf_map_delete_batch(int fd, void *in_batch, void *out_batch, + __u32 *count, __u64 elem_flags, + __u64 flags); +LIBBPF_API int bpf_map_lookup_batch(int fd, void *in_batch, void *out_batch, + void *keys, void *values, __u32 *count, + __u64 elem_flags, __u64 flags); +LIBBPF_API int bpf_map_lookup_and_delete_batch(int fd, void *in_batch, + void *out_batch, void *keys, + void *values, __u32 *count, + __u64 elem_flags, __u64 flags); +LIBBPF_API int bpf_map_update_batch(int fd, void *keys, void *values, + __u32 *count, __u64 elem_flags, + __u64 flags); + LIBBPF_API int bpf_obj_pin(int fd, const char *pathname); LIBBPF_API int bpf_obj_get(const char *pathname); LIBBPF_API int bpf_prog_attach(int prog_fd, int attachable_fd, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 8ddc2c40e482d..56462fea66f74 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -207,4 +207,8 @@ LIBBPF_0.0.6 { bpf_program__size; btf__find_by_name_kind; libbpf_find_vmlinux_btf_id; + bpf_map_delete_batch; + bpf_map_lookup_and_delete_batch; + bpf_map_lookup_batch; + bpf_map_update_batch; } LIBBPF_0.0.5; From patchwork Tue Nov 19 19:30:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197605 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="Cmi03rmw"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXW3xQKz9sPv for ; Wed, 20 Nov 2019 06:31:07 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727512AbfKSTbF (ORCPT ); Tue, 19 Nov 2019 14:31:05 -0500 Received: from mail-pg1-f201.google.com ([209.85.215.201]:51859 "EHLO mail-pg1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727535AbfKSTbF (ORCPT ); Tue, 19 Nov 2019 14:31:05 -0500 Received: by mail-pg1-f201.google.com with SMTP id f18so4026231pgh.18 for ; Tue, 19 Nov 2019 11:31:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=UVq6jpQg/LNjDNZAINhBK5BhAgNJ+pc6Cyp4zXHQ5bk=; b=Cmi03rmwd//plD6T6OyZ0hvhNFrDL0J81Qra4TZilKuIiud/Eals/ds9ajIhcTHzpa 0rxRyib9k8QYpS9iOCxXnZ4IggyuYyaP+/EeSyaUNDiDg1HRPl7FPkeBbFzOkwOYfwi0 /wd28w4jzMYma6uMVcQXiOrwJVw9j7diDhfocjonEK6wEmJBoFFDS0o58GTZPGvPtq/v rydVrgkLMErJQAIZ1Wy54HCK4PUSkICy3Hqu39g9iv/2u5t4DTJZuTMPRyf4UH0vfBUw 66Ocf89l+zmLYxIouwso9+flH4N6osHha3P5GdmKW3qRieepVOtgn0JS6Um4Z5Lu/5N/ zHFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UVq6jpQg/LNjDNZAINhBK5BhAgNJ+pc6Cyp4zXHQ5bk=; b=LxGrOuvT4Y/U8BdYdfuUzEB+k3iI5aTyzfqhStBkQvXjmXe6EHGX7CrgDogEoMTnez rl7l2VGD9biFN0W0/lT5NXA49Q0YKJJM1X6o9OEm6TTmchDq/5YysgozapiUcPXCU+VZ jcj2YiEkk+Ma//AXReYMg1wxLzsbKPcZh75dCSOGNlexiwBYFS5q6FgZY752g3b3LRHB cB3sN01DHs1bP8PxXrs8l/bu7fu6QoDFbnAp+1AX/46Pz3SIGR7IgX41D/C700NLVC/n 72azCAzmN85XDzpTdRuYtOsysJBoVZAS4MJmzykinyGEjnsKDL/R5ribaPgSWjxV9MvB gFlg== X-Gm-Message-State: APjAAAUk6HzE2KP82j7jERpkTWS0093P+h5eOmN2etg/ODnvWbDglJGw hiRt4t6FEqRrRt8Ltx9EfUI198CB1W6i X-Google-Smtp-Source: APXvYqzM02kO6RhwHi8KQwUiwrhvbj3RsvoKVxBf7q0qYIiqEuz5qZ8IOVzl8y6qv6pOLHl2zSvlGy7/tsP5 X-Received: by 2002:a63:a804:: with SMTP id o4mr6874477pgf.401.1574191863504; Tue, 19 Nov 2019 11:31:03 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:35 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-9-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 8/9] selftests/bpf: add batch ops testing for hmap and hmap_percpu From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Yonghong Song Tested bpf_map_lookup_and_delete_batch() and bpf_map_update_batch() functionality. $ ./test_maps ... test_hmap_lookup_and_delete_batch:PASS test_pcpu_hmap_lookup_and_delete_batch:PASS ... Signed-off-by: Yonghong Song Signed-off-by: Brian Vazquez --- .../map_lookup_and_delete_batch_htab.c | 257 ++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100644 tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_htab.c diff --git a/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_htab.c b/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_htab.c new file mode 100644 index 0000000000000..93e024cb85c60 --- /dev/null +++ b/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_htab.c @@ -0,0 +1,257 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include +#include +#include + +#include +#include + +#include +#include + +static void map_batch_update(int map_fd, __u32 max_entries, int *keys, + void *values, bool is_pcpu) +{ + typedef BPF_DECLARE_PERCPU(int, value); + int i, j, err; + value *v; + + if (is_pcpu) + v = (value *)values; + + for (i = 0; i < max_entries; i++) { + keys[i] = i + 1; + if (is_pcpu) + for (j = 0; j < bpf_num_possible_cpus(); j++) + bpf_percpu(v[i], j) = i + 2 + j; + else + ((int *)values)[i] = i + 2; + } + + err = bpf_map_update_batch(map_fd, keys, values, &max_entries, 0, 0); + CHECK(err, "bpf_map_update_batch()", "error:%s\n", strerror(errno)); +} + +static void map_batch_verify(int *visited, __u32 max_entries, + int *keys, void *values, bool is_pcpu) +{ + typedef BPF_DECLARE_PERCPU(int, value); + value *v; + int i, j; + + if (is_pcpu) + v = (value *)values; + + memset(visited, 0, max_entries * sizeof(*visited)); + for (i = 0; i < max_entries; i++) { + + if (is_pcpu) { + for (j = 0; j < bpf_num_possible_cpus(); j++) { + CHECK(keys[i] + 1 + j != bpf_percpu(v[i], j), + "key/value checking", + "error: i %d j %d key %d value %d\n", + i, j, keys[i], bpf_percpu(v[i], j)); + } + } else { + CHECK(keys[i] + 1 != ((int *)values)[i], + "key/value checking", + "error: i %d key %d value %d\n", i, keys[i], + ((int *)values)[i]); + } + + visited[i] = 1; + + } + for (i = 0; i < max_entries; i++) { + CHECK(visited[i] != 1, "visited checking", + "error: keys array at index %d missing\n", i); + } +} + +void __test_map_lookup_and_delete_batch(bool is_pcpu) +{ + int map_type = is_pcpu ? BPF_MAP_TYPE_PERCPU_HASH : BPF_MAP_TYPE_HASH; + struct bpf_create_map_attr xattr = { + .name = "hash_map", + .map_type = map_type, + .key_size = sizeof(int), + .value_size = sizeof(int), + }; + typedef BPF_DECLARE_PERCPU(int, value); + int map_fd, *keys, *visited, key; + __u32 batch = 0, count, total, total_success; + const __u32 max_entries = 10; + int err, i, step, value_size; + value pcpu_values[10]; + bool nospace_err; + void *values; + + xattr.max_entries = max_entries; + map_fd = bpf_create_map_xattr(&xattr); + CHECK(map_fd == -1, + "bpf_create_map_xattr()", "error:%s\n", strerror(errno)); + + value_size = is_pcpu ? sizeof(value) : sizeof(int); + keys = malloc(max_entries * sizeof(int)); + if (is_pcpu) + values = pcpu_values; + else + values = malloc(max_entries * sizeof(int)); + visited = malloc(max_entries * sizeof(int)); + CHECK(!keys || !values || !visited, "malloc()", + "error:%s\n", strerror(errno)); + + /* test 1: lookup/delete an empty hash table, -ENOENT */ + count = max_entries; + err = bpf_map_lookup_and_delete_batch(map_fd, NULL, &batch, keys, + values, &count, 0, 0); + CHECK((err && errno != ENOENT), "empty map", + "error: %s\n", strerror(errno)); + + /* populate elements to the map */ + map_batch_update(map_fd, max_entries, keys, values, is_pcpu); + + /* test 2: lookup/delete with count = 0, success */ + batch = 0; + count = 0; + err = bpf_map_lookup_and_delete_batch(map_fd, NULL, &batch, keys, + values, &count, 0, 0); + CHECK(err, "count = 0", "error: %s\n", strerror(errno)); + + /* test 3: lookup/delete with count = max_entries, success */ + memset(keys, 0, max_entries * sizeof(*keys)); + memset(values, 0, max_entries * value_size); + count = max_entries; + batch = 0; + err = bpf_map_lookup_and_delete_batch(map_fd, NULL, &batch, keys, + values, &count, 0, 0); + CHECK((err && errno != ENOENT), "count = max_entries", + "error: %s\n", strerror(errno)); + CHECK(count != max_entries, "count = max_entries", + "count = %u, max_entries = %u\n", count, max_entries); + map_batch_verify(visited, max_entries, keys, values, is_pcpu); + + /* bpf_map_get_next_key() should return -ENOENT for an empty map. */ + err = bpf_map_get_next_key(map_fd, NULL, &key); + CHECK(!err, "bpf_map_get_next_key()", "error: %s\n", strerror(errno)); + + /* test 4: lookup/delete in a loop with various steps. */ + total_success = 0; + for (step = 1; step < max_entries; step++) { + map_batch_update(map_fd, max_entries, keys, values, is_pcpu); + memset(keys, 0, max_entries * sizeof(*keys)); + memset(values, 0, max_entries * value_size); + batch = 0; + total = 0; + i = 0; + /* iteratively lookup/delete elements with 'step' + * elements each + */ + count = step; + nospace_err = false; + while (true) { + err = bpf_map_lookup_batch(map_fd, + total ? &batch : NULL, + &batch, keys + total, + values + + total * value_size, + &count, 0, 0); + /* It is possible that we are failing due to buffer size + * not big enough. In such cases, let us just exit and + * go with large steps. Not that a buffer size with + * max_entries should always work. + */ + if (err && errno == ENOSPC) { + nospace_err = true; + break; + } + + + CHECK((err && errno != ENOENT), "lookup with steps", + "error: %s\n", strerror(errno)); + + total += count; + if (err) + break; + + i++; + } + if (nospace_err == true) + continue; + + CHECK(total != max_entries, "lookup with steps", + "total = %u, max_entries = %u\n", total, max_entries); + map_batch_verify(visited, max_entries, keys, values, is_pcpu); + + memset(keys, 0, max_entries * sizeof(*keys)); + memset(values, 0, max_entries * value_size); + batch = 0; + total = 0; + i = 0; + /* iteratively lookup/delete elements with 'step' + * elements each + */ + count = step; + nospace_err = false; + while (true) { + err = bpf_map_lookup_and_delete_batch(map_fd, + total ? &batch : NULL, + &batch, keys + total, + values + + total * value_size, + &count, 0, 0); + /* It is possible that we are failing due to buffer size + * not big enough. In such cases, let us just exit and + * go with large steps. Not that a buffer size with + * max_entries should always work. + */ + if (err && errno == ENOSPC) { + nospace_err = true; + break; + } + + CHECK((err && errno != ENOENT), "lookup with steps", + "error: %s\n", strerror(errno)); + + total += count; + if (err) + break; + i++; + } + + if (nospace_err == true) + continue; + + CHECK(total != max_entries, "lookup/delete with steps", + "total = %u, max_entries = %u\n", total, max_entries); + + map_batch_verify(visited, max_entries, keys, values, is_pcpu); + err = bpf_map_get_next_key(map_fd, NULL, &key); + CHECK(!err, "bpf_map_get_next_key()", "error: %s\n", + strerror(errno)); + + total_success++; + } + + CHECK(total_success == 0, "check total_success", + "unexpected failure\n"); +} + +void test_hmap_lookup_and_delete_batch(void) +{ + __test_map_lookup_and_delete_batch(false); + printf("%s:PASS\n", __func__); +} + +void test_pcpu_hmap_lookup_and_delete_batch(void) +{ + __test_map_lookup_and_delete_batch(true); + printf("%s:PASS\n", __func__); +} + +void test_map_lookup_and_delete_batch_htab(void) +{ + test_hmap_lookup_and_delete_batch(); + test_pcpu_hmap_lookup_and_delete_batch(); +} From patchwork Tue Nov 19 19:30:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Vazquez X-Patchwork-Id: 1197607 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="fD0abG5c"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47HbXc2Mvkz9sPv for ; Wed, 20 Nov 2019 06:31:12 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727626AbfKSTbJ (ORCPT ); Tue, 19 Nov 2019 14:31:09 -0500 Received: from mail-pl1-f201.google.com ([209.85.214.201]:56968 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727578AbfKSTbH (ORCPT ); Tue, 19 Nov 2019 14:31:07 -0500 Received: by mail-pl1-f201.google.com with SMTP id x11so13671499plr.23 for ; Tue, 19 Nov 2019 11:31:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vOGdWKkLZw+fnwsKztc5Xhx6sgiytcIM3uQq3wSpoPY=; b=fD0abG5cMWtSthF8awD7gX4qx8kK59gTADR0EBrVD3nGRiZz614u+uZbjqr8zK9cVB x/tmy4rL3RFbOn3vTIO0GRrW4g86Py4+f3LKH9m5b79UdcN7Hu3WOK6fJO3gsjGNPpoD ok32+N3PNMkNiHCZm3rWij8ThooSDjZL2xjqebbx/yezL5DUA54NVw2GSC1kyurUylUj 51ZCEs2ivNGVNgWr7CWru9Aj9aKlZ05iUWE+P+HrutQITPGAJKiKvqWCU7RRLWZnInDM 2MiNXX9sCX7QezWEE9D92nEoDLZ3HjlzLG8thxkQB15LGo9PG1F0XlHljRipxC/8sS0g W+YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vOGdWKkLZw+fnwsKztc5Xhx6sgiytcIM3uQq3wSpoPY=; b=DuiXJCeu6fN0FlXG8DvMWT4av7fhFI2QWmH784mWt4UdFtjDNSMB3DuZWbK6FH6TuV WIevt5fZWjmgdxCsI+d0GppvNV6xe29iTMvZ9CZibaPv31j6pH03+YYAtnYMK+flEDoH IUPfvDRl6NWL1dmjQkhpzg6T9LrTAr+Iiao9m2/8RCyOv86kQgBntdNergRjI2YxxBqa UKW5xX6sxIdhR1bTuw8nUY4VNse1QvCHPqP3vBdCh46T3Q1JEcHl4ZMu7mYhC/WLQ867 10zR1tRrNik8X8+fuU2Bw+zwmkM87YvDOTosFykQ8+QuSNg6gxjUho535gMC1QgJyoKA 6VBQ== X-Gm-Message-State: APjAAAVJ6FJhsGrf/LWdayJAwt0VKls92aSVZHPxjsKRnczfxFuglNln Tsr449a91wvC/itxfnngVL1h5IrMoHdL X-Google-Smtp-Source: APXvYqyso4pIsRCIKj+mbEI/YKn/Jx2s4xp+YSqDX8EfM1n3NV90GXmXRfHtR/JpSzL7iTSU3F76PyGumonF X-Received: by 2002:a63:596:: with SMTP id 144mr7644456pgf.207.1574191866162; Tue, 19 Nov 2019 11:31:06 -0800 (PST) Date: Tue, 19 Nov 2019 11:30:36 -0800 In-Reply-To: <20191119193036.92831-1-brianvv@google.com> Message-Id: <20191119193036.92831-10-brianvv@google.com> Mime-Version: 1.0 References: <20191119193036.92831-1-brianvv@google.com> X-Mailer: git-send-email 2.24.0.432.g9d3f5f5b63-goog Subject: [PATCH v2 bpf-next 9/9] selftests/bpf: add batch ops testing to array bpf map From: Brian Vazquez To: Brian Vazquez , Alexei Starovoitov , Daniel Borkmann , "David S . Miller" Cc: Yonghong Song , Stanislav Fomichev , Petar Penkov , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Brian Vazquez Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Tested bpf_map_lookup_batch() and bpf_map_update_batch() functionality. $ ./test_maps ... test_map_lookup_and_delete_batch_array:PASS ... Signed-off-by: Brian Vazquez Signed-off-by: Yonghong Song --- .../map_lookup_and_delete_batch_array.c | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_array.c diff --git a/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_array.c b/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_array.c new file mode 100644 index 0000000000000..cbec72ad38609 --- /dev/null +++ b/tools/testing/selftests/bpf/map_tests/map_lookup_and_delete_batch_array.c @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include + +#include +#include + +#include + +static void map_batch_update(int map_fd, __u32 max_entries, int *keys, + int *values) +{ + int i, err; + + for (i = 0; i < max_entries; i++) { + keys[i] = i; + values[i] = i + 1; + } + + err = bpf_map_update_batch(map_fd, keys, values, &max_entries, 0, 0); + CHECK(err, "bpf_map_update_batch()", "error:%s\n", strerror(errno)); +} + +static void map_batch_verify(int *visited, __u32 max_entries, + int *keys, int *values) +{ + int i; + + memset(visited, 0, max_entries * sizeof(*visited)); + for (i = 0; i < max_entries; i++) { + CHECK(keys[i] + 1 != values[i], "key/value checking", + "error: i %d key %d value %d\n", i, keys[i], values[i]); + visited[i] = 1; + } + for (i = 0; i < max_entries; i++) { + CHECK(visited[i] != 1, "visited checking", + "error: keys array at index %d missing\n", i); + } +} + +void test_map_lookup_and_delete_batch_array(void) +{ + struct bpf_create_map_attr xattr = { + .name = "array_map", + .map_type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(int), + .value_size = sizeof(int), + }; + int map_fd, *keys, *values, *visited; + __u32 count, total, total_success; + const __u32 max_entries = 10; + int err, i, step; + bool nospace_err; + __u64 batch = 0; + + xattr.max_entries = max_entries; + map_fd = bpf_create_map_xattr(&xattr); + CHECK(map_fd == -1, + "bpf_create_map_xattr()", "error:%s\n", strerror(errno)); + + keys = malloc(max_entries * sizeof(int)); + values = malloc(max_entries * sizeof(int)); + visited = malloc(max_entries * sizeof(int)); + CHECK(!keys || !values || !visited, "malloc()", "error:%s\n", + strerror(errno)); + + /* populate elements to the map */ + map_batch_update(map_fd, max_entries, keys, values); + + /* test 1: lookup in a loop with various steps. */ + total_success = 0; + for (step = 1; step < max_entries; step++) { + map_batch_update(map_fd, max_entries, keys, values); + memset(keys, 0, max_entries * sizeof(*keys)); + memset(values, 0, max_entries * sizeof(*values)); + batch = 0; + total = 0; + i = 0; + /* iteratively lookup/delete elements with 'step' + * elements each. + */ + count = step; + nospace_err = false; + while (true) { + err = bpf_map_lookup_batch(map_fd, + total ? &batch : NULL, &batch, + keys + total, + values + total, + &count, 0, 0); + + CHECK((err && errno != ENOENT), "lookup with steps", + "error: %s\n", strerror(errno)); + + total += count; + + if (err) + break; + + i++; + } + + if (nospace_err == true) + continue; + + CHECK(total != max_entries, "lookup with steps", + "total = %u, max_entries = %u\n", total, max_entries); + + map_batch_verify(visited, max_entries, keys, values); + + total_success++; + } + + CHECK(total_success == 0, "check total_success", + "unexpected failure\n"); + + printf("%s:PASS\n", __func__); +}