| Message ID | 4D92190F.8050904@canonical.com |
|---|---|
| State | New |
| Headers | show |
On 03/29/2011 11:38 AM, Stefan Bader wrote: > On 03/29/2011 05:13 PM, Stefan Bader wrote: >> On 03/29/2011 04:51 PM, Tim Gardner wrote: >>> On 03/28/2011 02:05 AM, Stefan Bader wrote: >>>> On 03/25/2011 03:43 PM, John Johansen wrote: >>>>> On 03/25/2011 07:36 AM, Tim Gardner wrote: >>>>>> On 03/25/2011 08:30 AM, John Johansen wrote: >>>>>>> On 03/25/2011 06:49 AM, Tim Gardner wrote: >>>>>>>> On 03/25/2011 07:16 AM, John Johansen wrote: >>>>>>>> >>>>>>>>>> I'm still not convinced that CONFIG_NET_NS=n isn't the best >>>>>>>>>> solution, despite the complaints that change might elicit. I'd >>>>>>>>>> like to hear from the consumers of network name spaces about >>>>>>>>>> how they are using the feature, and possible workarounds if it >>>>>>>>>> were to go away. >>>>>>>>>> >>>>>>>>> That is the solution I would like but I think that at least for >>>>>>>>> the server that is going to be problematic. Container are seeing >>>>>>>>> a lot of use. >>>>>>>>> >>>>>>>> >>>>>>>> While containers in general are in use, are network name spaces >>>>>>>> pro-actively being used? Is there some workload that is _dependent_ >>>>>>>> on NET_NS ? I'm not proposing that we disable containers or other >>>>>>>> name space features, only NET_NS. >>>>>>>> >>>>>>> I don't know the answer to that, it is worth exploring. It is >>>>>>> pro-actively being used in that some applications are requesting the >>>>>>> CLONE_NEWNET flag, and I have seen container workloads that could >>>>>>> claim to require NET_NS (essentially replacing virtual machines with >>>>>>> containers) but I am not sure what kernel they were using. I >>>>>>> actually would like to turn NET_NS off too, my concern is that it is >>>>>>> a regression of the feature that some (unquantifiable) set of users >>>>>>> are using. >>>>>>> >>>>>>>>> If we were to go with an SRU of this I would lean towards the >>>>>>>>> smaller patchset that is enough to prevent memory being eaten (7 >>>>>>>>> of the 13), and then if speed is a problem the remain 6 could be >>>>>>>>> SRUed afterwards. >>>>>>>> >>>>>>>> I'm not keen on releasing a kernel that reduces connection >>>>>>>> setup/teardown by an order of magnitude. Surely this'll have an >>>>>>>> adverse impact on web servers and the like. >>>>>>>> >>>>>>> Neither am I but, but my perhaps flawed understanding was it should >>>>>>> only affect the connection setup/teardown if a new network namespace >>>>>>> is being created, and I doubt most use cases actually do this. This >>>>>>> is actually something we should get a better handle on, what work >>>>>>> loads that use NET_NS are noticeably impacted by this. >>>>>> >>>>>> Well, there is an alternative for those folks that _are_ dependent on NET_NS: >>>>>> >>>>>> sudo apt-get install linux-image-server-lts-backport-maverick >>>>>> >>>>> oh right, that convinces me. Turn it off. >>>> >>>> Sorry for stepping back into this discussion late. I think as long as it ok to >>>> go with a Maverick kernel, this clearly would be better than any partial >>>> backport for NET_NS. At least in the one specific case it was suggested that >>>> doing this was (at least for the moment) not possible as Maverick caused other >>>> problems. There was however only very vague hints about what those other >>>> problems really are, so it is hard to tell whether and how those could be solved. >>>> >>>> Just for the record, it is my understanding (as John already mentioned) that the >>>> slowdown only affects connections made with network namspace cloning involved. >>>> If that feature is not used, there should be no real slowdown. >>>> >>>> Summarizing, I think the safest solution is to turn the feature off. But we need >>>> to be careful about it. This being a LTS release there could also be people >>>> using it and for various reasons (sw cert) not be able to move to a newer kernel >>>> that simple. Unfortunately it is hard to find out before doing. >>>> >>>> -Stefan >>> >>> So, are you gonna send a patch disabling NET_NS ? >>> >> Oh well, yeah. Knowingly break something just scares me. But I'll send something >> out and let the stable guys decide. :-P >> >> -Stefan >> > So this would be the change to turn them off. > Acked-by: Tim Gardner <tim.gardner@canonical.com>
On 03/29/2011 10:38 AM, Stefan Bader wrote: > On 03/29/2011 05:13 PM, Stefan Bader wrote: >> On 03/29/2011 04:51 PM, Tim Gardner wrote: >>> On 03/28/2011 02:05 AM, Stefan Bader wrote: >>>> On 03/25/2011 03:43 PM, John Johansen wrote: >>>>> On 03/25/2011 07:36 AM, Tim Gardner wrote: >>>>>> On 03/25/2011 08:30 AM, John Johansen wrote: >>>>>>> On 03/25/2011 06:49 AM, Tim Gardner wrote: >>>>>>>> On 03/25/2011 07:16 AM, John Johansen wrote: >>>>>>>> >>>>>>>>>> I'm still not convinced that CONFIG_NET_NS=n isn't the best >>>>>>>>>> solution, despite the complaints that change might elicit. I'd >>>>>>>>>> like to hear from the consumers of network name spaces about >>>>>>>>>> how they are using the feature, and possible workarounds if it >>>>>>>>>> were to go away. >>>>>>>>>> >>>>>>>>> That is the solution I would like but I think that at least for >>>>>>>>> the server that is going to be problematic. Container are seeing >>>>>>>>> a lot of use. >>>>>>>>> >>>>>>>> >>>>>>>> While containers in general are in use, are network name spaces >>>>>>>> pro-actively being used? Is there some workload that is _dependent_ >>>>>>>> on NET_NS ? I'm not proposing that we disable containers or other >>>>>>>> name space features, only NET_NS. >>>>>>>> >>>>>>> I don't know the answer to that, it is worth exploring. It is >>>>>>> pro-actively being used in that some applications are requesting the >>>>>>> CLONE_NEWNET flag, and I have seen container workloads that could >>>>>>> claim to require NET_NS (essentially replacing virtual machines with >>>>>>> containers) but I am not sure what kernel they were using. I >>>>>>> actually would like to turn NET_NS off too, my concern is that it is >>>>>>> a regression of the feature that some (unquantifiable) set of users >>>>>>> are using. >>>>>>> >>>>>>>>> If we were to go with an SRU of this I would lean towards the >>>>>>>>> smaller patchset that is enough to prevent memory being eaten (7 >>>>>>>>> of the 13), and then if speed is a problem the remain 6 could be >>>>>>>>> SRUed afterwards. >>>>>>>> >>>>>>>> I'm not keen on releasing a kernel that reduces connection >>>>>>>> setup/teardown by an order of magnitude. Surely this'll have an >>>>>>>> adverse impact on web servers and the like. >>>>>>>> >>>>>>> Neither am I but, but my perhaps flawed understanding was it should >>>>>>> only affect the connection setup/teardown if a new network namespace >>>>>>> is being created, and I doubt most use cases actually do this. This >>>>>>> is actually something we should get a better handle on, what work >>>>>>> loads that use NET_NS are noticeably impacted by this. >>>>>> >>>>>> Well, there is an alternative for those folks that _are_ dependent on NET_NS: >>>>>> >>>>>> sudo apt-get install linux-image-server-lts-backport-maverick >>>>>> >>>>> oh right, that convinces me. Turn it off. >>>> >>>> Sorry for stepping back into this discussion late. I think as long as it ok to >>>> go with a Maverick kernel, this clearly would be better than any partial >>>> backport for NET_NS. At least in the one specific case it was suggested that >>>> doing this was (at least for the moment) not possible as Maverick caused other >>>> problems. There was however only very vague hints about what those other >>>> problems really are, so it is hard to tell whether and how those could be solved. >>>> >>>> Just for the record, it is my understanding (as John already mentioned) that the >>>> slowdown only affects connections made with network namspace cloning involved. >>>> If that feature is not used, there should be no real slowdown. >>>> >>>> Summarizing, I think the safest solution is to turn the feature off. But we need >>>> to be careful about it. This being a LTS release there could also be people >>>> using it and for various reasons (sw cert) not be able to move to a newer kernel >>>> that simple. Unfortunately it is hard to find out before doing. >>>> >>>> -Stefan >>> >>> So, are you gonna send a patch disabling NET_NS ? >>> >> Oh well, yeah. Knowingly break something just scares me. But I'll send something >> out and let the stable guys decide. :-P >> >> -Stefan >> > So this would be the change to turn them off. > Acked-by: Brad Figg <brad.figg@canonical.com>
On 03/29/2011 11:38 AM, Stefan Bader wrote: > On 03/29/2011 05:13 PM, Stefan Bader wrote: >> On 03/29/2011 04:51 PM, Tim Gardner wrote: >>> On 03/28/2011 02:05 AM, Stefan Bader wrote: >>>> On 03/25/2011 03:43 PM, John Johansen wrote: >>>>> On 03/25/2011 07:36 AM, Tim Gardner wrote: >>>>>> On 03/25/2011 08:30 AM, John Johansen wrote: >>>>>>> On 03/25/2011 06:49 AM, Tim Gardner wrote: >>>>>>>> On 03/25/2011 07:16 AM, John Johansen wrote: >>>>>>>> >>>>>>>>>> I'm still not convinced that CONFIG_NET_NS=n isn't the best >>>>>>>>>> solution, despite the complaints that change might elicit. I'd >>>>>>>>>> like to hear from the consumers of network name spaces about >>>>>>>>>> how they are using the feature, and possible workarounds if it >>>>>>>>>> were to go away. >>>>>>>>>> >>>>>>>>> That is the solution I would like but I think that at least for >>>>>>>>> the server that is going to be problematic. Container are seeing >>>>>>>>> a lot of use. >>>>>>>>> >>>>>>>> >>>>>>>> While containers in general are in use, are network name spaces >>>>>>>> pro-actively being used? Is there some workload that is _dependent_ >>>>>>>> on NET_NS ? I'm not proposing that we disable containers or other >>>>>>>> name space features, only NET_NS. >>>>>>>> >>>>>>> I don't know the answer to that, it is worth exploring. It is >>>>>>> pro-actively being used in that some applications are requesting the >>>>>>> CLONE_NEWNET flag, and I have seen container workloads that could >>>>>>> claim to require NET_NS (essentially replacing virtual machines with >>>>>>> containers) but I am not sure what kernel they were using. I >>>>>>> actually would like to turn NET_NS off too, my concern is that it is >>>>>>> a regression of the feature that some (unquantifiable) set of users >>>>>>> are using. >>>>>>> >>>>>>>>> If we were to go with an SRU of this I would lean towards the >>>>>>>>> smaller patchset that is enough to prevent memory being eaten (7 >>>>>>>>> of the 13), and then if speed is a problem the remain 6 could be >>>>>>>>> SRUed afterwards. >>>>>>>> >>>>>>>> I'm not keen on releasing a kernel that reduces connection >>>>>>>> setup/teardown by an order of magnitude. Surely this'll have an >>>>>>>> adverse impact on web servers and the like. >>>>>>>> >>>>>>> Neither am I but, but my perhaps flawed understanding was it should >>>>>>> only affect the connection setup/teardown if a new network namespace >>>>>>> is being created, and I doubt most use cases actually do this. This >>>>>>> is actually something we should get a better handle on, what work >>>>>>> loads that use NET_NS are noticeably impacted by this. >>>>>> >>>>>> Well, there is an alternative for those folks that _are_ dependent on NET_NS: >>>>>> >>>>>> sudo apt-get install linux-image-server-lts-backport-maverick >>>>>> >>>>> oh right, that convinces me. Turn it off. >>>> >>>> Sorry for stepping back into this discussion late. I think as long as it ok to >>>> go with a Maverick kernel, this clearly would be better than any partial >>>> backport for NET_NS. At least in the one specific case it was suggested that >>>> doing this was (at least for the moment) not possible as Maverick caused other >>>> problems. There was however only very vague hints about what those other >>>> problems really are, so it is hard to tell whether and how those could be solved. >>>> >>>> Just for the record, it is my understanding (as John already mentioned) that the >>>> slowdown only affects connections made with network namspace cloning involved. >>>> If that feature is not used, there should be no real slowdown. >>>> >>>> Summarizing, I think the safest solution is to turn the feature off. But we need >>>> to be careful about it. This being a LTS release there could also be people >>>> using it and for various reasons (sw cert) not be able to move to a newer kernel >>>> that simple. Unfortunately it is hard to find out before doing. >>>> >>>> -Stefan >>> >>> So, are you gonna send a patch disabling NET_NS ? >>> >> Oh well, yeah. Knowingly break something just scares me. But I'll send something >> out and let the stable guys decide. :-P >> >> -Stefan >> > So this would be the change to turn them off. > applied, pushed
From 903091e6af33e8f336fd78e9d9ab178d0761856e Mon Sep 17 00:00:00 2001 From: Stefan Bader <stefan.bader@canonical.com> Date: Tue, 29 Mar 2011 19:26:56 +0200 Subject: [PATCH] UBUNTU: (config) Disable CONFIG_NET_NS BugLink: http://bugs.launchpad.net/bugs/720095 While network namespaces are used for containers, the implementation was still experimental in Lucid. Even worse the cleanup of network namespaces is very slow. This can cause severe memory issues when an application uses them a lot (for example when cloning subprocesses). The problem can only be avoided by backporting several upstream patches (and without even more changes that will slow down the creation of new net namespaces). This cannot really be justified as a stable update. So turn the option off in Lucid. Users of containers should be pointed to the LTS-backport kernel. Signed-off-by: Stefan Bader <stefan.bader@canonical.com> --- debian.master/config/config.common.ports | 2 +- debian.master/config/config.common.ubuntu | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/debian.master/config/config.common.ports b/debian.master/config/config.common.ports index 9173ecd..82c9c88 100644 --- a/debian.master/config/config.common.ports +++ b/debian.master/config/config.common.ports @@ -2092,7 +2092,7 @@ CONFIG_NET_IPGRE_BROADCAST=y CONFIG_NET_IPIP=m CONFIG_NET_KEY=m # CONFIG_NET_KEY_MIGRATE is not set -CONFIG_NET_NS=y +# CONFIG_NET_NS is not set CONFIG_NET_PCI=y CONFIG_NET_PCMCIA=y CONFIG_NET_POCKET=y diff --git a/debian.master/config/config.common.ubuntu b/debian.master/config/config.common.ubuntu index 03d8049..8a93892 100644 --- a/debian.master/config/config.common.ubuntu +++ b/debian.master/config/config.common.ubuntu @@ -2532,7 +2532,7 @@ CONFIG_NET_IPIP=m CONFIG_NET_ISA=y CONFIG_NET_KEY=m # CONFIG_NET_KEY_MIGRATE is not set -CONFIG_NET_NS=y +# CONFIG_NET_NS is not set CONFIG_NET_PCI=y CONFIG_NET_PCMCIA=y CONFIG_NET_PKTGEN=m -- 1.7.0.4