From patchwork Thu Aug 28 19:44:22 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Kirby X-Patchwork-Id: 384010 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 9B50C1400B6 for ; Fri, 29 Aug 2014 05:45:01 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752252AbaH1Toa (ORCPT ); Thu, 28 Aug 2014 15:44:30 -0400 Received: from peace.netnation.com ([204.174.223.2]:46117 "EHLO peace.netnation.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752057AbaH1To3 (ORCPT ); Thu, 28 Aug 2014 15:44:29 -0400 Received: from sim by peace.netnation.com with local (Exim 4.80) (envelope-from ) id 1XN5cM-000564-SC; Thu, 28 Aug 2014 12:44:22 -0700 Date: Thu, 28 Aug 2014 12:44:22 -0700 From: Simon Kirby To: "Paul E. McKenney" Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, "Eric W. Biederman" Subject: Re: net_ns cleanup / RCU overhead Message-ID: <20140828194422.GB8867@hostway.ca> References: <20140820055855.GB5579@hostway.ca> <20140828192431.GF5001@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20140828192431.GF5001@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu, Aug 28, 2014 at 12:24:31PM -0700, Paul E. McKenney wrote: > On Tue, Aug 19, 2014 at 10:58:55PM -0700, Simon Kirby wrote: > > Hello! > > > > In trying to figure out what happened to a box running lots of vsftpd > > since we deployed a CONFIG_NET_NS=y kernel to it, we found that the > > (wall) time needed for cleanup_net() to complete, even on an idle box, > > can be quite long: > > > > #!/bin/bash > > > > ip netns delete test >&/dev/null > > while ip netns add test; do > > echo hi > > ip netns delete test > > done > > > > On my desktop and typical hosts, this prints at only around 4 or 6 per > > second. While this is happening, "vmstat 1" reports 100% idle, and there > > there are D-state processes with stacks similar to: > > > > 30566 [kworker/u16:1] D wait_rcu_gp+0x48, synchronize_sched+0x2f, cleanup_net+0xdb, process_one_work+0x175, worker_thread+0x119, kthread+0xbb, ret_from_fork+0x7c, 0xffffffffffffffff > > > > 32220 ip D copy_net_ns+0x68, create_new_namespaces+0xfc, unshare_nsproxy_namespaces+0x66, SyS_unshare+0x159, system_call_fastpath+0x16, 0xffffffffffffffff > > > > copy_net_ns() is waiting on net_mutex which is held by cleanup_net(). > > > > vsftpd uses CLONE_NEWNET to set up privsep processes. There is a comment > > about it being really slow before 2.6.35 (it avoids CLONE_NEWNET in that > > case). I didn't find anything that makes 2.6.35 any faster, but on Debian > > 2.6.36-5-amd64, I notice it does seem to be a bit faster than 3.2, 3.10, > > 3.16, though still not anything I'd ever want to rely on per connection. > > > > C implementation of the above: http://0x.ca/sim/ref/tools/netnsloop.c > > > > Kernel stack "top": http://0x.ca/sim/ref/tools/pstack > > > > What's going on here? > > That is a bit slow for many configurations, but there are some exceptions. > > So, what is your kernel's .config? I was unable to find a config (or stock kernel) that was any different, but here's the one we're using: http://0x.ca/sim/ref/3.10/config-3.10.53 How fast does the above test run for you? We've been running with the attached, which has helped a little, but it's still quite slow in our particular use case (vsftpd), and with the above test. Should I enable RCU_TRACE or STALL_INFO with a low timeout or something? Simon- -- >8 -- Subject: [PATCH] netns: use synchronize_rcu_expedited instead of synchronize_rcu Similar to ef323088, with synchronize_rcu(), we are only able to create and destroy about 4 or 7 net namespaces per second, which really puts a dent in the performance of programs attempting to use CLONE_NEWNET for privilege separation (vsftpd, chromium). --- net/core/net_namespace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index 85b6269..6dcb4b3 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -296,7 +296,7 @@ static void cleanup_net(struct work_struct *work) * This needs to be before calling the exit() notifiers, so * the rcu_barrier() below isn't sufficient alone. */ - synchronize_rcu(); + synchronize_rcu_expedited(); /* Run all of the network namespace exit methods */ list_for_each_entry_reverse(ops, &pernet_list, list)