| Message ID | 817e43c5-d88d-e616-7074-5715de29d319@Netapp.com |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | David Miller |
| Headers | show |
On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote: > >-----Original Message----- > >From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] > >Sent: Wednesday, October 26, 2016 5:40 PM > >To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; Andy > >Adamson <andros@netapp.com>; Anna Schumaker > ><Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org > >Cc: netdev@vger.kernel.org; Trond Myklebust <Trond.Myklebust@netapp.com>; > >Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com> > >Subject: Re: nfs NULL-dereferencing in net-next > > > >On 10/25/2016 01:19 PM, Yotam Gigi wrote: > >> > >>> -----Original Message----- > >>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] > >On > >>> Behalf Of Jakub Kicinski > >>> Sent: Monday, October 17, 2016 10:20 PM > >>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker > >>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org > >>> Cc: netdev@vger.kernel.org; Trond Myklebust > ><Trond.Myklebust@netapp.com> > >>> Subject: nfs NULL-dereferencing in net-next > >>> > >>> Hi! > >>> > >>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f > >>> ("fsl/fman: fix error return code in mac_probe()"). > >> > >> > >> I see the same thing. It happens constantly on some of my machines, making > >them > >> completely unusable. > >> > >> I bisected it and got to the commit: > >> > >> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274 > >> Author: Andy Adamson <andros@netapp.com> > >> Date: Fri Sep 9 09:22:27 2016 -0400 > >> > >> NFS add xprt switch addrs test to match client > >> > >> Signed-off-by: Andy Adamson <andros@netapp.com> > >> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com> > > > >Thanks for reporting on this everyone! Does this patch help? > > Actually, I still see the same bug with the same trace. I rebuild the latest net-next and I'm not seeing the trace any more... I'm only seeing this (with or without your patch): [ 23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 [ 23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 [ 23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 [ 23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 [ 23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 [ 23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 HTH
On 10/26/2016 02:08 PM, Jakub Kicinski wrote: > On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote: >>> -----Original Message----- >>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] >>> Sent: Wednesday, October 26, 2016 5:40 PM >>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; Andy >>> Adamson <andros@netapp.com>; Anna Schumaker >>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>> Cc: netdev@vger.kernel.org; Trond Myklebust <Trond.Myklebust@netapp.com>; >>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com> >>> Subject: Re: nfs NULL-dereferencing in net-next >>> >>> On 10/25/2016 01:19 PM, Yotam Gigi wrote: >>>> >>>>> -----Original Message----- >>>>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] >>> On >>>>> Behalf Of Jakub Kicinski >>>>> Sent: Monday, October 17, 2016 10:20 PM >>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker >>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>>>> Cc: netdev@vger.kernel.org; Trond Myklebust >>> <Trond.Myklebust@netapp.com> >>>>> Subject: nfs NULL-dereferencing in net-next >>>>> >>>>> Hi! >>>>> >>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f >>>>> ("fsl/fman: fix error return code in mac_probe()"). >>>> >>>> >>>> I see the same thing. It happens constantly on some of my machines, making >>> them >>>> completely unusable. >>>> >>>> I bisected it and got to the commit: >>>> >>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274 >>>> Author: Andy Adamson <andros@netapp.com> >>>> Date: Fri Sep 9 09:22:27 2016 -0400 >>>> >>>> NFS add xprt switch addrs test to match client >>>> >>>> Signed-off-by: Andy Adamson <andros@netapp.com> >>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com> >>> >>> Thanks for reporting on this everyone! Does this patch help? >> >> Actually, I still see the same bug with the same trace. Well, it was worth a shot. I'll keep poking at it. > > I rebuild the latest net-next and I'm not seeing the trace any more... > I'm only seeing this (with or without your patch): > > [ 23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > [ 23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > [ 23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > [ 23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > [ 23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > [ 23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 Interesting, I get that too when I try to use NFS v4.1. It's weird that the crash would stop happening like that, so maybe something is racy in this area. Thanks for testing, Yotam and Jakub! Anna > > HTH >
>-----Original Message----- >From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] >Sent: Wednesday, October 26, 2016 9:17 PM >To: Jakub Kicinski <kubakici@wp.pl> >Cc: Yotam Gigi <yotamg@mellanox.com>; Andy Adamson <andros@netapp.com>; >linux-nfs@vger.kernel.org; netdev@vger.kernel.org; Trond Myklebust ><Trond.Myklebust@netapp.com>; Yotam Gigi <yotam.gi@gmail.com>; mlxsw ><mlxsw@mellanox.com> >Subject: Re: nfs NULL-dereferencing in net-next > >On 10/26/2016 02:08 PM, Jakub Kicinski wrote: >> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote: >>>> -----Original Message----- >>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] >>>> Sent: Wednesday, October 26, 2016 5:40 PM >>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; >Andy >>>> Adamson <andros@netapp.com>; Anna Schumaker >>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>>> Cc: netdev@vger.kernel.org; Trond Myklebust ><Trond.Myklebust@netapp.com>; >>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com> >>>> Subject: Re: nfs NULL-dereferencing in net-next >>>> >>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote: >>>>> >>>>>> -----Original Message----- >>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev- >owner@vger.kernel.org] >>>> On >>>>>> Behalf Of Jakub Kicinski >>>>>> Sent: Monday, October 17, 2016 10:20 PM >>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker >>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust >>>> <Trond.Myklebust@netapp.com> >>>>>> Subject: nfs NULL-dereferencing in net-next >>>>>> >>>>>> Hi! >>>>>> >>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f >>>>>> ("fsl/fman: fix error return code in mac_probe()"). >>>>> >>>>> >>>>> I see the same thing. It happens constantly on some of my machines, making >>>> them >>>>> completely unusable. >>>>> >>>>> I bisected it and got to the commit: >>>>> >>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274 >>>>> Author: Andy Adamson <andros@netapp.com> >>>>> Date: Fri Sep 9 09:22:27 2016 -0400 >>>>> >>>>> NFS add xprt switch addrs test to match client >>>>> >>>>> Signed-off-by: Andy Adamson <andros@netapp.com> >>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com> >>>> >>>> Thanks for reporting on this everyone! Does this patch help? >>> >>> Actually, I still see the same bug with the same trace. > >Well, it was worth a shot. I'll keep poking at it. > >> >> I rebuild the latest net-next and I'm not seeing the trace any more... >> I'm only seeing this (with or without your patch): >> >> [ 23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >> [ 23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >> [ 23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >> [ 23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >> [ 23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >> [ 23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >Interesting, I get that too when I try to use NFS v4.1. It's weird that the crash would >stop happening like that, so maybe something is racy in this area. > >Thanks for testing, Yotam and Jakub! >Anna I just found out that it happens on any of my machines, once I put two nfs entries in my fstab. If I put only one, I don't see the problem. I hope it might be helpful :) > >> >> HTH >>
On Thu, 27 Oct 2016 06:50:22 +0000, Yotam Gigi wrote: > >-----Original Message----- > >From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] > >Sent: Wednesday, October 26, 2016 9:17 PM > >To: Jakub Kicinski <kubakici@wp.pl> > >Cc: Yotam Gigi <yotamg@mellanox.com>; Andy Adamson <andros@netapp.com>; > >linux-nfs@vger.kernel.org; netdev@vger.kernel.org; Trond Myklebust > ><Trond.Myklebust@netapp.com>; Yotam Gigi <yotam.gi@gmail.com>; mlxsw > ><mlxsw@mellanox.com> > >Subject: Re: nfs NULL-dereferencing in net-next > > > >On 10/26/2016 02:08 PM, Jakub Kicinski wrote: > >> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote: > >>>> -----Original Message----- > >>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] > >>>> Sent: Wednesday, October 26, 2016 5:40 PM > >>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; > >Andy > >>>> Adamson <andros@netapp.com>; Anna Schumaker > >>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org > >>>> Cc: netdev@vger.kernel.org; Trond Myklebust > ><Trond.Myklebust@netapp.com>; > >>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com> > >>>> Subject: Re: nfs NULL-dereferencing in net-next > >>>> > >>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote: > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev- > >owner@vger.kernel.org] > >>>> On > >>>>>> Behalf Of Jakub Kicinski > >>>>>> Sent: Monday, October 17, 2016 10:20 PM > >>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker > >>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org > >>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust > >>>> <Trond.Myklebust@netapp.com> > >>>>>> Subject: nfs NULL-dereferencing in net-next > >>>>>> > >>>>>> Hi! > >>>>>> > >>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f > >>>>>> ("fsl/fman: fix error return code in mac_probe()"). > >>>>> > >>>>> > >>>>> I see the same thing. It happens constantly on some of my machines, making > >>>> them > >>>>> completely unusable. > >>>>> > >>>>> I bisected it and got to the commit: > >>>>> > >>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274 > >>>>> Author: Andy Adamson <andros@netapp.com> > >>>>> Date: Fri Sep 9 09:22:27 2016 -0400 > >>>>> > >>>>> NFS add xprt switch addrs test to match client > >>>>> > >>>>> Signed-off-by: Andy Adamson <andros@netapp.com> > >>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com> > >>>> > >>>> Thanks for reporting on this everyone! Does this patch help? > >>> > >>> Actually, I still see the same bug with the same trace. > > > >Well, it was worth a shot. I'll keep poking at it. > > > >> > >> I rebuild the latest net-next and I'm not seeing the trace any more... > >> I'm only seeing this (with or without your patch): > >> > >> [ 23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >> [ 23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >> [ 23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >> [ 23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >> [ 23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > >> [ 23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 > > > >Interesting, I get that too when I try to use NFS v4.1. It's weird that the crash would > >stop happening like that, so maybe something is racy in this area. > > > >Thanks for testing, Yotam and Jakub! > >Anna > > I just found out that it happens on any of my machines, once I put two nfs entries in > my fstab. If I put only one, I don't see the problem. > > I hope it might be helpful :) Hi Anna, any updates on this one? The crash came back half an hour after I reported that it was gone... Over the weekend David Miller rebased net-next on top of 4.9.0-rc3 and the bug is still there :( FWIW I also have multiple nfs mounts on my setup, 2 in fstab and one in a startup script. Following Yotam I dropped one of the fstab entries and things seem to be working (even though I still have multiple mounts, the other one just comes a bit later).
On 10/31/2016 11:25 AM, Jakub Kicinski wrote: > On Thu, 27 Oct 2016 06:50:22 +0000, Yotam Gigi wrote: >>> -----Original Message----- >>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] >>> Sent: Wednesday, October 26, 2016 9:17 PM >>> To: Jakub Kicinski <kubakici@wp.pl> >>> Cc: Yotam Gigi <yotamg@mellanox.com>; Andy Adamson <andros@netapp.com>; >>> linux-nfs@vger.kernel.org; netdev@vger.kernel.org; Trond Myklebust >>> <Trond.Myklebust@netapp.com>; Yotam Gigi <yotam.gi@gmail.com>; mlxsw >>> <mlxsw@mellanox.com> >>> Subject: Re: nfs NULL-dereferencing in net-next >>> >>> On 10/26/2016 02:08 PM, Jakub Kicinski wrote: >>>> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote: >>>>>> -----Original Message----- >>>>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com] >>>>>> Sent: Wednesday, October 26, 2016 5:40 PM >>>>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; >>> Andy >>>>>> Adamson <andros@netapp.com>; Anna Schumaker >>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust >>> <Trond.Myklebust@netapp.com>; >>>>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com> >>>>>> Subject: Re: nfs NULL-dereferencing in net-next >>>>>> >>>>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote: >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev- >>> owner@vger.kernel.org] >>>>>> On >>>>>>>> Behalf Of Jakub Kicinski >>>>>>>> Sent: Monday, October 17, 2016 10:20 PM >>>>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker >>>>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org >>>>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust >>>>>> <Trond.Myklebust@netapp.com> >>>>>>>> Subject: nfs NULL-dereferencing in net-next >>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f >>>>>>>> ("fsl/fman: fix error return code in mac_probe()"). >>>>>>> >>>>>>> >>>>>>> I see the same thing. It happens constantly on some of my machines, making >>>>>> them >>>>>>> completely unusable. >>>>>>> >>>>>>> I bisected it and got to the commit: >>>>>>> >>>>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274 >>>>>>> Author: Andy Adamson <andros@netapp.com> >>>>>>> Date: Fri Sep 9 09:22:27 2016 -0400 >>>>>>> >>>>>>> NFS add xprt switch addrs test to match client >>>>>>> >>>>>>> Signed-off-by: Andy Adamson <andros@netapp.com> >>>>>>> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com> >>>>>> >>>>>> Thanks for reporting on this everyone! Does this patch help? >>>>> >>>>> Actually, I still see the same bug with the same trace. >>> >>> Well, it was worth a shot. I'll keep poking at it. >>> >>>> >>>> I rebuild the latest net-next and I'm not seeing the trace any more... >>>> I'm only seeing this (with or without your patch): >>>> >>>> [ 23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>>> [ 23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>>> [ 23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>>> [ 23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>>> [ 23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>>> [ 23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0 >>> >>> Interesting, I get that too when I try to use NFS v4.1. It's weird that the crash would >>> stop happening like that, so maybe something is racy in this area. >>> >>> Thanks for testing, Yotam and Jakub! >>> Anna >> >> I just found out that it happens on any of my machines, once I put two nfs entries in >> my fstab. If I put only one, I don't see the problem. >> >> I hope it might be helpful :) > > Hi Anna, > > any updates on this one? The crash came back half an hour after I > reported that it was gone... Aww, I was hoping that patch would work. It still seemed to fix some issues for me when mounting multiple servers, so I'm planning to keep it. Unfortunately I'm out of town this week, so I haven't had much of a chance to keep poking at this issue. I should be able to get back to it next week! Thanks for the update! Anna > > Over the weekend David Miller rebased net-next on top of 4.9.0-rc3 and > the bug is still there :( FWIW I also have multiple nfs mounts on my > setup, 2 in fstab and one in a startup script. Following Yotam I > dropped one of the fstab entries and things seem to be working (even > though I still have multiple mounts, the other one just comes a bit > later). >
On Thu, Nov 03, Anna Schumaker wrote: > Aww, I was hoping that patch would work. It still seemed to fix some > issues for me when mounting multiple servers, so I'm planning to keep > it. Unfortunately I'm out of town this week, so I haven't had much of > a chance to keep poking at this issue. I should be able to get back > to it next week! Is this supposed to be fixed already? I get an oops in rpc_clnt_xprt_switch_has_addr+0xc/0x40 with 4.9.0-rc4. Olaf
On 11/10/2016 09:47 AM, Olaf Hering wrote: > On Thu, Nov 03, Anna Schumaker wrote: > >> Aww, I was hoping that patch would work. It still seemed to fix some >> issues for me when mounting multiple servers, so I'm planning to keep >> it. Unfortunately I'm out of town this week, so I haven't had much of >> a chance to keep poking at this issue. I should be able to get back >> to it next week! > > Is this supposed to be fixed already? I get an oops in > rpc_clnt_xprt_switch_has_addr+0xc/0x40 with 4.9.0-rc4. Not yet, sorry. I'm waiting on one bugfix patch before sending my pull request. Thanks for waiting patiently, Anna > > Olaf >
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index 34dd7b2..62a4827 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -2753,14 +2753,18 @@ EXPORT_SYMBOL_GPL(rpc_cap_max_reconnect_timeout); void rpc_clnt_xprt_switch_put(struct rpc_clnt *clnt) { + rcu_read_lock(); xprt_switch_put(rcu_dereference(clnt->cl_xpi.xpi_xpswitch)); + rcu_read_unlock(); } EXPORT_SYMBOL_GPL(rpc_clnt_xprt_switch_put); void rpc_clnt_xprt_switch_add_xprt(struct rpc_clnt *clnt, struct rpc_xprt *xprt) { + rcu_read_lock(); rpc_xprt_switch_add_xprt(rcu_dereference(clnt->cl_xpi.xpi_xpswitch), xprt); + rcu_read_unlock(); } EXPORT_SYMBOL_GPL(rpc_clnt_xprt_switch_add_xprt); @@ -2770,9 +2774,8 @@ bool rpc_clnt_xprt_switch_has_addr(struct rpc_clnt *clnt, struct rpc_xprt_switch *xps; bool ret; - xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch); - rcu_read_lock(); + xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch); ret = rpc_xprt_switch_has_addr(xps, sap); rcu_read_unlock(); return ret;