From patchwork Fri Mar 20 11:40:22 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 24749 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by ozlabs.org (Postfix) with ESMTP id CB55EDDEE8 for ; Fri, 20 Mar 2009 23:03:58 +1100 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753227AbZCTMDU (ORCPT ); Fri, 20 Mar 2009 08:03:20 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752970AbZCTMDU (ORCPT ); Fri, 20 Mar 2009 08:03:20 -0400 Received: from gw1.cosmosbay.com ([212.99.114.194]:58283 "EHLO gw1.cosmosbay.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750999AbZCTMDT (ORCPT ); Fri, 20 Mar 2009 08:03:19 -0400 Received: from [127.0.0.1] (localhost [127.0.0.1]) by gw1.cosmosbay.com (8.13.7/8.13.7) with ESMTP id n2KBeMq0024512; Fri, 20 Mar 2009 12:40:22 +0100 Message-ID: <49C380A6.4000904@cosmosbay.com> Date: Fri, 20 Mar 2009 12:40:22 +0100 From: Eric Dumazet User-Agent: Thunderbird 2.0.0.21 (Windows/20090302) MIME-Version: 1.0 To: "David S. Miller" CC: Linux Netdev List Subject: [RFC] net: release dst entry in dev_queue_xmit() X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (gw1.cosmosbay.com [0.0.0.0]); Fri, 20 Mar 2009 12:40:22 +0100 (CET) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org One point of contention in high network loads is the dst_release() performed when a transmited skb is freed. This is because NIC tx completion calls skb free long after original call to dev_queue_xmit(skb). CPU cache is cold and the atomic op in dst_release() stalls. On SMP, this is quite visible if one CPU is 100% handling softirqs for a network device, since dst_clone() is done by other cpus, involving cache line ping pongs. I believe we can release dst in dev_queue_xmit(), while cpu cache is hot, since caller of dev_queue_xmit() had to hold a reference on dst right before. This reduce work to be done by softirq handler, and decrease cache misses. I also believe only pktgen can call dev_queue_xmit() with skb which have a skb->users != 1. But pkthen skbs have a NULL dst entry. I added a WARN_ON_ONCE() to catch other cases, and not release skb->dst if skb->users != 1 Signed-off-by: Eric Dumazet Acked-by: Neil Horman --- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/core/dev.c b/net/core/dev.c index f112970..9e0fd01 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1852,6 +1852,20 @@ gso: if (q->enqueue) { spinlock_t *root_lock = qdisc_lock(q); + /* + * Release dst while its refcnt is hot in CPU cache, instead + * of waiting NIC tx completion + */ + if (likely(skb->dst)) { + if (!WARN_ON_ONCE(atomic_read(&skb->users) != 1)) { + int newrefcnt; + + smp_mb__before_atomic_dec(); + newrefcnt = atomic_dec_return(&skb->dst->__refcnt); + WARN_ON(newrefcnt < 0); + skb->dst = NULL; + } + } spin_lock(root_lock); if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {