From patchwork Wed Dec 26 20:27:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018733 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ikXwQqk7"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4L73kXTz9sCr for ; Thu, 27 Dec 2018 07:28:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727819AbeLZU23 (ORCPT ); Wed, 26 Dec 2018 15:28:29 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:42757 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727689AbeLZU2Y (ORCPT ); Wed, 26 Dec 2018 15:28:24 -0500 Received: by mail-pf1-f196.google.com with SMTP id 64so8216756pfr.9 for ; Wed, 26 Dec 2018 12:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=5IK0vZiHxM5SDcSN+srCKjjCaT5uK7+dKJmAZ7vek3Y=; b=ikXwQqk7rG1LxucPFQgQxA3oigaOjl/VJXXKQXfETMtwXZPryr5GjkwXoHMazNCjiK Cs/k8DtszJ3XEhhWph7BxYsEdg8a/XiJcajxjEDu/6uGxujrhQsO/37zbkIr2OL1qSKV qMR4sSg/T3qmTAc+ww1Lxy1ITkXDC8xmoJGmYXcYYA/ZObfLRQJ1hf6nuPen49DIsPBN paQUDM83rzWr8YirHNR2Hub8XRj94rvP8CC/wEuABeTzbbWGEJMXUbTWa0xLV6NV+Cao 9y4dYtoOlS4NMRx8XV/r/A2TX5tQI0rxtt3c/MpeBoYuaca1J2rPHWs3aB0sHxkDt/kW KfxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=5IK0vZiHxM5SDcSN+srCKjjCaT5uK7+dKJmAZ7vek3Y=; b=RdiBM+vY7j7o3ZD37UJBl4FXepiu5TjWrCRW33AiraFRWallJLhoJquAaDO5+tZWe7 5nWGIApbIOkXrrd1KOBcsuMpyUwAi734Mq7mGs4K688MPl9xm9O0DzK501qSVzXJQTiJ S/dt3BU+fwzIjaZjxu1EwzBeX8LWQCtdP8ueT+rUiYNN49fC3yyk71GuEKZpaPfD0+n+ SaOsa3ObqmX1F/Xoy178r34mWfUWcZHLERd/HuSeXmNo7PsEUL9MnKo5HZubBIGWGGdS wVMKuH/AKiXb7lHUemYlUAq/S+oSUZaA0S9fYr0GjgV2/B55bXyNEVriVG9AYQVnh9ZX kUAg== X-Gm-Message-State: AA+aEWZ48BNf8GbfRjGXS7VCdZxKE3w9Z/nlPvxFE52NGdRJzORqoOFW 5A2387cdgJU6DNnevJVzrTFQFwaA X-Google-Smtp-Source: AFSGD/U87PsfUz3TeZfjsXGjLyLQ8IVqSdBocwJSh57G0jRNwI2z/CQRzH6pGMP2WbuDskumpcVCHQ== X-Received: by 2002:a62:a99:: with SMTP id 25mr21164077pfk.121.1545856103317; Wed, 26 Dec 2018 12:28:23 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:22 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 4/6] veth: add zero-copy AF_XDP TX support. Date: Wed, 26 Dec 2018 12:27:51 -0800 Message-Id: <1545856073-8680-5-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Remove the extra copy when doing AF_XDP TX. The xdp frame comes directly from the umem element and passes to the receiving logic. Also, only depending on async_xmit to kick napi poll isn't fast enough. So re-schedule the napi at the end of poll so the ksoftirqd can keep processing the packets. The performance increases from 1.1Mpps to 1.4Mpps, when running zero copy xdpsock as sender and XDP_DROP at the receiver side. Signed-off-by: William Tu --- drivers/net/veth.c | 41 ++++++++++++++++++----------------------- 1 file changed, 18 insertions(+), 23 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 10cf9ded59f1..551444195398 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -761,45 +761,34 @@ static int veth_xsk_poll(struct napi_struct *napi, int budget) while (peer_rq->xsk_umem && budget--) { unsigned int inner_xdp_xmit = 0; unsigned int metasize = 0; - struct xdp_frame *xdpf; + struct xdp_frame xdpf; bool dropped = false; struct sk_buff *skb; struct page *page; void *vaddr; - void *addr; u32 len; if (!xsk_umem_consume_tx_virtual(peer_rq->xsk_umem, &vaddr, &len)) break; - page = dev_alloc_page(); - if (!page) { - xsk_umem_complete_tx(peer_rq->xsk_umem, 1); - xsk_umem_consume_tx_done(peer_rq->xsk_umem); - return -ENOMEM; - } - - addr = page_to_virt(page); - xdpf = addr; - memset(xdpf, 0, sizeof(*xdpf)); - - addr += sizeof(*xdpf); - memcpy(addr, vaddr, len); + xdpf.data = vaddr + metasize; + xdpf.len = len; + xdpf.headroom = 0; + xdpf.metasize = metasize; + xdpf.mem.type = MEM_TYPE_ZERO_COPY_VDEV; - xdpf->data = addr + metasize; - xdpf->len = len; - xdpf->headroom = 0; - xdpf->metasize = metasize; - xdpf->mem.type = MEM_TYPE_PAGE_SHARED; + page = virt_to_head_page(vaddr); + if (page->mem_cgroup) + page->mem_cgroup = NULL; /* put into rq */ - skb = veth_xdp_rcv_one(rq, xdpf, &inner_xdp_xmit); + skb = veth_xdp_rcv_one(rq, &xdpf, &inner_xdp_xmit); if (!skb) { /* Peer side has XDP program attached */ if (inner_xdp_xmit & VETH_XDP_TX) { /* Not supported */ pr_warn("veth: peer XDP_TX not supported\n"); - xdp_return_frame(xdpf); + xdp_return_frame(&xdpf); dropped = true; goto skip_tx; } else if (inner_xdp_xmit & VETH_XDP_REDIR) { @@ -808,7 +797,8 @@ static int veth_xsk_poll(struct napi_struct *napi, int budget) dropped = true; } } else { - napi_gro_receive(&rq->xdp_napi, skb); + napi_gro_receive(&rq->xdp_napi, skb_copy(skb, GFP_KERNEL)); + kfree(skb); } skip_tx: xsk_umem_complete_tx(peer_rq->xsk_umem, 1); @@ -856,6 +846,11 @@ static int veth_poll(struct napi_struct *napi, int budget) xdp_do_flush_map(); xdp_clear_return_frame_no_direct(); + /* schedule again so the CPU can keep receiving + * at higher rate + */ + napi_schedule(&rq->xdp_napi); + return done > budget ? budget : done; }