From patchwork Tue Apr 16 12:58:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Magnus Karlsson X-Patchwork-Id: 1086292 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44k55z3Y7Mz9s4Y for ; Tue, 16 Apr 2019 22:58:47 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727902AbfDPM6r (ORCPT ); Tue, 16 Apr 2019 08:58:47 -0400 Received: from mga05.intel.com ([192.55.52.43]:29568 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727240AbfDPM6q (ORCPT ); Tue, 16 Apr 2019 08:58:46 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Apr 2019 05:58:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,357,1549958400"; d="scan'208";a="292005246" Received: from mkarlsso-mobl.ger.corp.intel.com (HELO localhost.localdomain) ([10.249.47.235]) by orsmga004.jf.intel.com with ESMTP; 16 Apr 2019 05:58:42 -0700 From: Magnus Karlsson To: magnus.karlsson@intel.com, bjorn.topel@intel.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org Cc: bpf@vger.kernel.org, bruce.richardson@intel.com, ciara.loftus@intel.com, ilias.apalodimas@linaro.org, xiaolong.ye@intel.com, ferruh.yigit@intel.com, qi.z.zhang@intel.com, georgmueller@gmx.net Subject: [PATCH bpf v3 5/5] libbpf: optimize barrier for XDP socket rings Date: Tue, 16 Apr 2019 14:58:12 +0200 Message-Id: <1555419492-16906-6-git-send-email-magnus.karlsson@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1555419492-16906-1-git-send-email-magnus.karlsson@intel.com> References: <1555419492-16906-1-git-send-email-magnus.karlsson@intel.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org The full memory barrier in the XDP socket rings on the consumer side between the load of the data and the store of the consumer ring is there to protect the store from being executed before the load of the data. If this was allowed to happen, the producer might overwrite the data field with a new entry before the consumer got the chance to read it. On x86, stores are guaranteed not to be reordered with older loads, so it does not need a full memory barrier here. A compile time barrier would be enough. This patch introdcues a new primitive in libbpf_util.h that implements a new barrier type (libbpf_smp_rwmb) hindering stores to be reordered with older loads. It is then used in the XDP socket ring access code in libbpf to improve performance. Signed-off-by: Magnus Karlsson --- tools/lib/bpf/libbpf_util.h | 5 +++++ tools/lib/bpf/xsk.h | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/libbpf_util.h b/tools/lib/bpf/libbpf_util.h index 71fbb28..172b707e 100644 --- a/tools/lib/bpf/libbpf_util.h +++ b/tools/lib/bpf/libbpf_util.h @@ -32,20 +32,25 @@ do { \ # define libbpf_smp_wmb() asm volatile("" : : : "memory") # define libbpf_smp_mb() \ asm volatile("lock; addl $0,-4(%%rsp)" : : : "memory", "cc") +/* Hinders stores to be observed before older loads. */ +# define libbpf_smp_rwmb() asm volatile("" : : : "memory") #elif defined(__aarch64__) # define libbpf_smp_rmb() asm volatile("dmb ishld" : : : "memory") # define libbpf_smp_wmb() asm volatile("dmb ishst" : : : "memory") # define libbpf_smp_mb() asm volatile("dmb ish" : : : "memory") +# define libbpf_smp_rwmb() libbpf_smp_mb() #elif defined(__arm__) /* These are only valid for armv7 and above */ # define libbpf_smp_rmb() asm volatile("dmb ish" : : : "memory") # define libbpf_smp_wmb() asm volatile("dmb ishst" : : : "memory") # define libbpf_smp_mb() asm volatile("dmb ish" : : : "memory") +# define libbpf_smp_rwmb() libbpf_smp_mb() #else # warning Architecture missing native barrier functions in libbpf_util.h. # define libbpf_smp_rmb() __sync_synchronize() # define libbpf_smp_wmb() __sync_synchronize() # define libbpf_smp_mb() __sync_synchronize() +# define libbpf_smp_rwmb() __sync_synchronize() #endif #ifdef __cplusplus diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h index 2377c7a..82ea71a 100644 --- a/tools/lib/bpf/xsk.h +++ b/tools/lib/bpf/xsk.h @@ -152,7 +152,7 @@ static inline void xsk_ring_cons__release(struct xsk_ring_cons *cons, size_t nb) /* Make sure data has been read before indicating we are done * with the entries by updating the consumer pointer. */ - libbpf_smp_mb(); + libbpf_smp_rwmb(); *cons->consumer += nb; }