Message ID | 1251239734.3169.65.camel@w-sridhar.beaverton.ibm.com |
---|---|
State | RFC, archived |
Delegated to: | David Miller |
Headers | show |
On Tue, 25 Aug 2009, Sridhar Samudrala wrote: > Could you check if the UDP packet losses you are seeing are accounted for in > qdisc drops with this patch. But i am not completely positive on this as this > case happens only if qdisc is deactivated. This does not work. qdisc drops are still not reported. They are reported for IP and UDP. Test tool crashes on first TX overrun: clameter@rd-strategy3-deb64:~$ ./mcast -n1 -r400000 Receiver: Listening to control channel 239.0.192.1 Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0 origin 10.2.36.123 Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1 sec. sendto: No buffer space available Socket Send error netstat reports exactly one packet loss: clameter@rd-strategy3-deb64:~$ netstat -su IcmpMsg: InType3: 1 OutType3: 1 Udp: 298 packets received 0 packets to unknown port received. 0 packet receive errors 7232136 packets sent SndbufErrors: 1 root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 SNMP report one drop: root@rd-strategy3-deb64:/home/clameter#cat /proc/net/snmp Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates Ip: 2 64 1114 0 0 0 0 0 1114 7232754 1 0 0 0 0 0 0 0 0 Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps Icmp: 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 IcmpMsg: InType3 OutType3 IcmpMsg: 1 1 Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts Tcp: 1 200 120000 -1 26 4 0 0 2 774 595 0 0 0 Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors Udp: 308 0 0 7232146 0 1 UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors UdpLite: 0 0 0 0 0 0 -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 2009-08-26 at 12:29 -0400, Christoph Lameter wrote: > On Tue, 25 Aug 2009, Sridhar Samudrala wrote: > > > Could you check if the UDP packet losses you are seeing are accounted for in > > qdisc drops with this patch. But i am not completely positive on this as this > > case happens only if qdisc is deactivated. > > This does not work. qdisc drops are still not reported. OK. So the drops are not happening in dev_queue_xmit(). > They are reported for IP and UDP. Not clear what you meant by this. > Test tool crashes on first TX overrun: > > clameter@rd-strategy3-deb64:~$ ./mcast -n1 -r400000 > Receiver: Listening to control channel 239.0.192.1 > Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0 > origin 10.2.36.123 > Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1 > sec. > sendto: No buffer space available > Socket Send error > > netstat reports exactly one packet loss: > > > clameter@rd-strategy3-deb64:~$ netstat -su > IcmpMsg: > InType3: 1 > OutType3: 1 > Udp: > 298 packets received > 0 packets to unknown port received. > 0 packet receive errors > 7232136 packets sent > SndbufErrors: 1 > > root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show > qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 > 1 1 1 1 > Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0) > rate 0bit 0pps backlog 0b 0p requeues 0 Even the Sent count seems to be too low. Are you looking at the right device? So based on the current analysis, the packets are getting dropped after the call to ip_local_out() in ip_push_pending_frames(). ip_local_out() is failing with NET_XMIT_DROP. But we are not sure where they are getting dropped. Is that right? I think we need to figure out where they are getting dropped and then decide on the appropriate counter to be incremented. Thanks Sridhar -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 26 Aug 2009, Sridhar Samudrala wrote: > > They are reported for IP and UDP. > Not clear what you meant by this. The SNMP and UDP statistics show the loss. qdisc level does not show the loss. > > root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show > > qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 > > 1 1 1 1 > > Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0) > > rate 0bit 0pps backlog 0b 0p requeues 0 > > Even the Sent count seems to be too low. Are you looking at the right > device? I would think that tc displays all queues? It says eth0 and eth0 is the device that we sent the data out on. > So based on the current analysis, the packets are getting dropped after > the call to ip_local_out() in ip_push_pending_frames(). ip_local_out() > is failing with NET_XMIT_DROP. But we are not sure where they are > getting dropped. Is that right? ip_local_out is returning ENOBUFS. Something at the qdisc layer is dropping the packet and not incrementing counters. > I think we need to figure out where they are getting dropped and then > decide on the appropriate counter to be incremented. Right. Where in the qdisc layer do drops occur? -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 2009-08-26 at 15:09 -0400, Christoph Lameter wrote: > On Wed, 26 Aug 2009, Sridhar Samudrala wrote: > > > > They are reported for IP and UDP. > > Not clear what you meant by this. > > The SNMP and UDP statistics show the loss. qdisc level does not show the > loss. > > > root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show > > > qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 > > > 1 1 1 1 > > > Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0) > > > rate 0bit 0pps backlog 0b 0p requeues 0 > > > > Even the Sent count seems to be too low. Are you looking at the right > > device? > > I would think that tc displays all queues? It says eth0 and eth0 is the > device that we sent the data out on. > > > So based on the current analysis, the packets are getting dropped after > > the call to ip_local_out() in ip_push_pending_frames(). ip_local_out() > > is failing with NET_XMIT_DROP. But we are not sure where they are > > getting dropped. Is that right? > > ip_local_out is returning ENOBUFS. Something at the qdisc layer is > dropping the packet and not incrementing counters. Is the ENOBUFS return with your/Eric's patch? I thought you were were seeing NET_XMIT_DROP without any patches. > > > I think we need to figure out where they are getting dropped and then > > decide on the appropriate counter to be incremented. > > Right. Where in the qdisc layer do drops occur? The normal path where the packets are dropped when the tx qlen is exceeded is pfifo_fast_enqueue() -> qdisc_drop() In this path, drops are counted. The other place is in dev_queue_xmit(), but you are not hitting that case too. So it looks like there is another place where they are getting dropped. Thanks Sridhar -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 26 Aug 2009, Sridhar Samudrala wrote: > > ip_local_out is returning ENOBUFS. Something at the qdisc layer is > > dropping the packet and not incrementing counters. > > Is the ENOBUFS return with your/Eric's patch? I thought you were > were seeing NET_XMIT_DROP without any patches. Both Erics latest patch and your patch were applied. > > > I think we need to figure out where they are getting dropped and then > > > decide on the appropriate counter to be incremented. > > > > Right. Where in the qdisc layer do drops occur? > > The normal path where the packets are dropped when the tx qlen is exceeded is > pfifo_fast_enqueue() -> qdisc_drop() > In this path, drops are counted. > The other place is in dev_queue_xmit(), but you are not hitting that case too. > > So it looks like there is another place where they are getting dropped. Hmmm.. I need to find more time to dig into this. Anyways it seems that Eric's latest patch is doing many good things for packet loss accounting. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/net/core/dev.c b/net/core/dev.c index 6a94475..8b6a075 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1864,8 +1864,7 @@ gso: spin_lock(root_lock); if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) { - kfree_skb(skb); - rc = NET_XMIT_DROP; + rc = qdisc_drop(skb, q); } else { rc = qdisc_enqueue_root(skb, q); qdisc_run(q);