Message ID | CALDO+SZ5CJfUuEzGUQNJhyfCQVrN98_Gg-v=JoO7eSFf4wCNBA@mail.gmail.com |
---|---|
State | Not Applicable |
Headers | show |
Hi William, could you try this fix? (I'm getting hard failures on those OVN tests on my dev box, likely something unrelated..) http://patchwork.ozlabs.org/patch/622833/ On 14 May 2016 at 17:13, William Tu <u9012063@gmail.com> wrote: > Hi Ben, > > Thanks, I applied diff below. Unfortunately it makes no difference. > > --- a/ofproto/ofproto-dpif-upcall.c > +++ b/ofproto/ofproto-dpif-upcall.c > @@ -402,7 +402,7 @@ udpif_create(struct dpif_backer *backer, struct dpif > *dpif) > atomic_init(&udpif->n_flows, 0); > atomic_init(&udpif->n_flows_timestamp, LLONG_MIN); > ovs_mutex_init(&udpif->n_flows_mutex); > - udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); > + udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); > for (int i = 0; i < N_UMAPS; i++) { > cmap_init(&udpif->ukeys[i].cmap); > > Regards, > William > > > On Sat, May 14, 2016 at 12:07 PM, Ben Pfaff <blp@ovn.org> wrote: > >> That's a really weird one. >> >> This is for an unusual code path: it's an OpenFlow packet-out, that gets >> executed via dpif_netdev, which causes recirculation, which installs a >> new datapath flow. I wonder whether somehow this bypasses a normal >> initialization step through some weirdness. >> >> Does it make any difference if you change >> udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); >> to >> udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); >> e.g. xmalloc -> xzalloc, in udpif_create()? That might give us >> some information. >> >> Thanks, >> >> Ben. >> >> On Fri, May 13, 2016 at 06:58:49PM -0700, William Tu wrote: >> > Hi, >> > >> > Valgrind testcase 2029: ovn -- 3 HVs, 3 LS, 3 lports/LS, 1 LR reports >> > acquiring an uninitialized mutex_lock at ovs_mutex_lock(&umap->mutex). It >> > seems that udpif_create() already initializes all the mutex_lock of >> > udpif->ukeys. I have no clue about how to solve it. Any comments are >> > appreciated. >> > >> > Command: ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif >> > -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file >> > >> > ==32048== Use of uninitialised value of size 8 >> > ==32048== at 0x4C023C: ovs_mutex_lock_at (ovs-thread.c:76) >> > ==32048== by 0x43486F: ukey_install_start (ofproto-dpif-upcall.c:1572) >> > ==32048== by 0x434BE7: ukey_install (ofproto-dpif-upcall.c:1634) >> > ==32048== by 0x434BE7: upcall_cb (ofproto-dpif-upcall.c:1174) >> > ==32048== by 0x458AF3: dp_netdev_upcall.isra.20 (dpif-netdev.c:3265) >> > ==32048== by 0x45CD90: fast_path_processing (dpif-netdev.c:3518) >> > ==32048== by 0x45D62F: dp_netdev_input__ (dpif-netdev.c:3623) >> > ==32048== by 0x45E06E: dp_netdev_recirculate (dpif-netdev.c:3647) >> > ==32048== by 0x45E06E: dp_execute_cb (dpif-netdev.c:3842) >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) >> > ==32048== by 0x45E94B: dp_netdev_execute_actions (dpif-netdev.c:3882) >> > ==32048== by 0x45E94B: dpif_netdev_execute (dpif-netdev.c:2368) >> > ==32048== by 0x45E94B: dpif_netdev_operate (dpif-netdev.c:2397) >> > ==32048== by 0x46043B: dpif_operate (dpif.c:1228) >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) >> > ==32048== by 0x460C91: dpif_execute (dpif.c:1185) >> > ==32048== by 0x460C91: dpif_execute_helper_cb (dpif.c:1132) >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) >> > ==32048== by 0x4603C1: dpif_execute_with_help (dpif.c:1169) >> > ==32048== by 0x4603C1: dpif_operate (dpif.c:1283) >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) >> > ==32048== by 0x428CE0: ofproto_dpif_execute_actions__ >> > (ofproto-dpif.c:3718) >> > ==32048== by 0x428DAA: ofproto_dpif_execute_actions >> (ofproto-dpif.c:3735) >> > ==32048== by 0x428DAA: packet_out (ofproto-dpif.c:4385) >> > ==32048== by 0x41C5E1: handle_packet_out (ofproto.c:3395) >> > ==32048== by 0x420B0A: handle_openflow__ (ofproto.c:7219) >> > ==32048== by 0x420B0A: handle_openflow (ofproto.c:7383) >> > ==32048== by 0x444D22: ofconn_run (connmgr.c:1376) >> > ==32048== by 0x444D22: connmgr_run (connmgr.c:320) >> > >> > Regards, >> > William >> > _______________________________________________ >> > dev mailing list >> > dev@openvswitch.org >> > http://openvswitch.org/mailman/listinfo/dev >> > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
Hi Joe, I did a quick try and the error is still there. Regards, William On Mon, May 16, 2016 at 8:09 PM, Joe Stringer <joe@ovn.org> wrote: > Hi William, could you try this fix? (I'm getting hard failures on > those OVN tests on my dev box, likely something unrelated..) > > http://patchwork.ozlabs.org/patch/622833/ > > > On 14 May 2016 at 17:13, William Tu <u9012063@gmail.com> wrote: > > Hi Ben, > > > > Thanks, I applied diff below. Unfortunately it makes no difference. > > > > --- a/ofproto/ofproto-dpif-upcall.c > > +++ b/ofproto/ofproto-dpif-upcall.c > > @@ -402,7 +402,7 @@ udpif_create(struct dpif_backer *backer, struct dpif > > *dpif) > > atomic_init(&udpif->n_flows, 0); > > atomic_init(&udpif->n_flows_timestamp, LLONG_MIN); > > ovs_mutex_init(&udpif->n_flows_mutex); > > - udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); > > + udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); > > for (int i = 0; i < N_UMAPS; i++) { > > cmap_init(&udpif->ukeys[i].cmap); > > > > Regards, > > William > > > > > > On Sat, May 14, 2016 at 12:07 PM, Ben Pfaff <blp@ovn.org> wrote: > > > >> That's a really weird one. > >> > >> This is for an unusual code path: it's an OpenFlow packet-out, that gets > >> executed via dpif_netdev, which causes recirculation, which installs a > >> new datapath flow. I wonder whether somehow this bypasses a normal > >> initialization step through some weirdness. > >> > >> Does it make any difference if you change > >> udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); > >> to > >> udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); > >> e.g. xmalloc -> xzalloc, in udpif_create()? That might give us > >> some information. > >> > >> Thanks, > >> > >> Ben. > >> > >> On Fri, May 13, 2016 at 06:58:49PM -0700, William Tu wrote: > >> > Hi, > >> > > >> > Valgrind testcase 2029: ovn -- 3 HVs, 3 LS, 3 lports/LS, 1 LR reports > >> > acquiring an uninitialized mutex_lock at > ovs_mutex_lock(&umap->mutex). It > >> > seems that udpif_create() already initializes all the mutex_lock of > >> > udpif->ukeys. I have no clue about how to solve it. Any comments are > >> > appreciated. > >> > > >> > Command: ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif > >> > -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > >> > > >> > ==32048== Use of uninitialised value of size 8 > >> > ==32048== at 0x4C023C: ovs_mutex_lock_at (ovs-thread.c:76) > >> > ==32048== by 0x43486F: ukey_install_start > (ofproto-dpif-upcall.c:1572) > >> > ==32048== by 0x434BE7: ukey_install (ofproto-dpif-upcall.c:1634) > >> > ==32048== by 0x434BE7: upcall_cb (ofproto-dpif-upcall.c:1174) > >> > ==32048== by 0x458AF3: dp_netdev_upcall.isra.20 > (dpif-netdev.c:3265) > >> > ==32048== by 0x45CD90: fast_path_processing (dpif-netdev.c:3518) > >> > ==32048== by 0x45D62F: dp_netdev_input__ (dpif-netdev.c:3623) > >> > ==32048== by 0x45E06E: dp_netdev_recirculate (dpif-netdev.c:3647) > >> > ==32048== by 0x45E06E: dp_execute_cb (dpif-netdev.c:3842) > >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) > >> > ==32048== by 0x45E94B: dp_netdev_execute_actions > (dpif-netdev.c:3882) > >> > ==32048== by 0x45E94B: dpif_netdev_execute (dpif-netdev.c:2368) > >> > ==32048== by 0x45E94B: dpif_netdev_operate (dpif-netdev.c:2397) > >> > ==32048== by 0x46043B: dpif_operate (dpif.c:1228) > >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) > >> > ==32048== by 0x460C91: dpif_execute (dpif.c:1185) > >> > ==32048== by 0x460C91: dpif_execute_helper_cb (dpif.c:1132) > >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) > >> > ==32048== by 0x4603C1: dpif_execute_with_help (dpif.c:1169) > >> > ==32048== by 0x4603C1: dpif_operate (dpif.c:1283) > >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) > >> > ==32048== by 0x428CE0: ofproto_dpif_execute_actions__ > >> > (ofproto-dpif.c:3718) > >> > ==32048== by 0x428DAA: ofproto_dpif_execute_actions > >> (ofproto-dpif.c:3735) > >> > ==32048== by 0x428DAA: packet_out (ofproto-dpif.c:4385) > >> > ==32048== by 0x41C5E1: handle_packet_out (ofproto.c:3395) > >> > ==32048== by 0x420B0A: handle_openflow__ (ofproto.c:7219) > >> > ==32048== by 0x420B0A: handle_openflow (ofproto.c:7383) > >> > ==32048== by 0x444D22: ofconn_run (connmgr.c:1376) > >> > ==32048== by 0x444D22: connmgr_run (connmgr.c:320) > >> > > >> > Regards, > >> > William > >> > _______________________________________________ > >> > dev mailing list > >> > dev@openvswitch.org > >> > http://openvswitch.org/mailman/listinfo/dev > >> > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > http://openvswitch.org/mailman/listinfo/dev >
Hi William, Could you see if this series fixes it? http://openvswitch.org/pipermail/dev/2016-May/071190.html Thanks, Daniele 2016-05-17 9:04 GMT-07:00 William Tu <u9012063@gmail.com>: > Hi Joe, > > I did a quick try and the error is still there. > > Regards, > William > > On Mon, May 16, 2016 at 8:09 PM, Joe Stringer <joe@ovn.org> wrote: > > > Hi William, could you try this fix? (I'm getting hard failures on > > those OVN tests on my dev box, likely something unrelated..) > > > > http://patchwork.ozlabs.org/patch/622833/ > > > > > > On 14 May 2016 at 17:13, William Tu <u9012063@gmail.com> wrote: > > > Hi Ben, > > > > > > Thanks, I applied diff below. Unfortunately it makes no difference. > > > > > > --- a/ofproto/ofproto-dpif-upcall.c > > > +++ b/ofproto/ofproto-dpif-upcall.c > > > @@ -402,7 +402,7 @@ udpif_create(struct dpif_backer *backer, struct > dpif > > > *dpif) > > > atomic_init(&udpif->n_flows, 0); > > > atomic_init(&udpif->n_flows_timestamp, LLONG_MIN); > > > ovs_mutex_init(&udpif->n_flows_mutex); > > > - udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); > > > + udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); > > > for (int i = 0; i < N_UMAPS; i++) { > > > cmap_init(&udpif->ukeys[i].cmap); > > > > > > Regards, > > > William > > > > > > > > > On Sat, May 14, 2016 at 12:07 PM, Ben Pfaff <blp@ovn.org> wrote: > > > > > >> That's a really weird one. > > >> > > >> This is for an unusual code path: it's an OpenFlow packet-out, that > gets > > >> executed via dpif_netdev, which causes recirculation, which installs a > > >> new datapath flow. I wonder whether somehow this bypasses a normal > > >> initialization step through some weirdness. > > >> > > >> Does it make any difference if you change > > >> udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); > > >> to > > >> udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); > > >> e.g. xmalloc -> xzalloc, in udpif_create()? That might give us > > >> some information. > > >> > > >> Thanks, > > >> > > >> Ben. > > >> > > >> On Fri, May 13, 2016 at 06:58:49PM -0700, William Tu wrote: > > >> > Hi, > > >> > > > >> > Valgrind testcase 2029: ovn -- 3 HVs, 3 LS, 3 lports/LS, 1 LR > reports > > >> > acquiring an uninitialized mutex_lock at > > ovs_mutex_lock(&umap->mutex). It > > >> > seems that udpif_create() already initializes all the mutex_lock of > > >> > udpif->ukeys. I have no clue about how to solve it. Any comments are > > >> > appreciated. > > >> > > > >> > Command: ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif > > >> > -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > > >> > > > >> > ==32048== Use of uninitialised value of size 8 > > >> > ==32048== at 0x4C023C: ovs_mutex_lock_at (ovs-thread.c:76) > > >> > ==32048== by 0x43486F: ukey_install_start > > (ofproto-dpif-upcall.c:1572) > > >> > ==32048== by 0x434BE7: ukey_install (ofproto-dpif-upcall.c:1634) > > >> > ==32048== by 0x434BE7: upcall_cb (ofproto-dpif-upcall.c:1174) > > >> > ==32048== by 0x458AF3: dp_netdev_upcall.isra.20 > > (dpif-netdev.c:3265) > > >> > ==32048== by 0x45CD90: fast_path_processing (dpif-netdev.c:3518) > > >> > ==32048== by 0x45D62F: dp_netdev_input__ (dpif-netdev.c:3623) > > >> > ==32048== by 0x45E06E: dp_netdev_recirculate (dpif-netdev.c:3647) > > >> > ==32048== by 0x45E06E: dp_execute_cb (dpif-netdev.c:3842) > > >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) > > >> > ==32048== by 0x45E94B: dp_netdev_execute_actions > > (dpif-netdev.c:3882) > > >> > ==32048== by 0x45E94B: dpif_netdev_execute (dpif-netdev.c:2368) > > >> > ==32048== by 0x45E94B: dpif_netdev_operate (dpif-netdev.c:2397) > > >> > ==32048== by 0x46043B: dpif_operate (dpif.c:1228) > > >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) > > >> > ==32048== by 0x460C91: dpif_execute (dpif.c:1185) > > >> > ==32048== by 0x460C91: dpif_execute_helper_cb (dpif.c:1132) > > >> > ==32048== by 0x4820EA: odp_execute_actions (odp-execute.c:533) > > >> > ==32048== by 0x4603C1: dpif_execute_with_help (dpif.c:1169) > > >> > ==32048== by 0x4603C1: dpif_operate (dpif.c:1283) > > >> > ==32048== by 0x460B37: dpif_execute.part.14 (dpif.c:1193) > > >> > ==32048== by 0x428CE0: ofproto_dpif_execute_actions__ > > >> > (ofproto-dpif.c:3718) > > >> > ==32048== by 0x428DAA: ofproto_dpif_execute_actions > > >> (ofproto-dpif.c:3735) > > >> > ==32048== by 0x428DAA: packet_out (ofproto-dpif.c:4385) > > >> > ==32048== by 0x41C5E1: handle_packet_out (ofproto.c:3395) > > >> > ==32048== by 0x420B0A: handle_openflow__ (ofproto.c:7219) > > >> > ==32048== by 0x420B0A: handle_openflow (ofproto.c:7383) > > >> > ==32048== by 0x444D22: ofconn_run (connmgr.c:1376) > > >> > ==32048== by 0x444D22: connmgr_run (connmgr.c:320) > > >> > > > >> > Regards, > > >> > William > > >> > _______________________________________________ > > >> > dev mailing list > > >> > dev@openvswitch.org > > >> > http://openvswitch.org/mailman/listinfo/dev > > >> > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > http://openvswitch.org/mailman/listinfo/dev > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev >
--- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -402,7 +402,7 @@ udpif_create(struct dpif_backer *backer, struct dpif *dpif) atomic_init(&udpif->n_flows, 0); atomic_init(&udpif->n_flows_timestamp, LLONG_MIN); ovs_mutex_init(&udpif->n_flows_mutex); - udpif->ukeys = xmalloc(N_UMAPS * sizeof *udpif->ukeys); + udpif->ukeys = xzalloc(N_UMAPS * sizeof *udpif->ukeys); for (int i = 0; i < N_UMAPS; i++) { cmap_init(&udpif->ukeys[i].cmap);