| Message ID | 1241435206.8115.104.camel@localhost.localdomain |
|---|---|
| State | Rejected, archived |
| Delegated to: | David Miller |
| Headers | show |
On Mon, May 4, 2009 at 4:06 AM, Jesper Dangaard Brouer <hawk@comx.dk> wrote: > > Hardware RX fifo overruns for the 82576 is stored in the > RNBC (Receive No Buffers Count) register. > > I choose the store the RNBC value in net_stats.rx_fifo_errors. > > Saving the stats in dev->net_stats makes it visible via > /proc/net/dev as "fifo", and thus viewable to ifconfig > as "overruns" and 'netstat -i' as "RX-OVR". > > The Receive No Buffers Count (RNBC) can already be queried by > ethtool -S as "rx_no_buffer_count". > > Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk> > --- > NAK. RNBC is not a counter for buffer overruns, and so should not be counted as such.
From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> Date: Tue, 5 May 2009 11:47:09 -0700 > NAK. RNBC is not a counter for buffer overruns, and so should not be > counted as such. I'd say technically it is, it indicates that more packets arrived than the available receive buffers could handle. If anything, this is the closest this device has for this kind of situation, and it's useful for diagnosing problems. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 5 May 2009, David Miller wrote: > From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > Date: Tue, 5 May 2009 11:47:09 -0700 > >> NAK. RNBC is not a counter for buffer overruns, and so should not be >> counted as such. > > I'd say technically it is, it indicates that more packets arrived than > the available receive buffers could handle. I agree with DaveM. Technically it _is_ a buffer overflow, but in the host memory not the NIC. I'm sort of pushing the system into a situation where it cannot empty the receive buffers fast enough. I can fairly easily provoke this situation by adding too many iptables rules, which (intentionally) cause high CPU load and causes ksoftirqd to run (I'm Oprofiling netfilter modules). > If anything, this is the closest this device has for this kind of > situation, and it's useful for diagnosing problems. Its really useful for diagnosing problems, and I'm betting that this is a real-life situation which people is going to experience. We might as well help our self to more easily identify this issue when people report drop problems. Notice that I'm seeing: rx_no_buffer_count: 136955 rx_missed_errors: 0 Thus, the rx_missed_errors is zero, which according to the datasheet is the "real" fifo drop (the MPC register, Missed Packets Count) and PCI bandwidth problem indications. If we really should nitpick, then: adapter->net_stats.rx_missed_errors = adapter->stats.mpc Should then have been stored in the rx_fifo_errors. Notice that rx_missed_errors is presented to userspace as drops (see net/core/dev.c:2624). I think that both MPC and RNBC should be stored in rx_fifo_errors (and of cause still keeping them seperate to ethtool -S). I'll post two patches with these changes tomorrow, for you evaluation. Please reconsider you NAK. Greetings, Jesper Brouer -- ------------------------------------------------------------------- MSc. Master of Computer Science Dept. of Computer Science, University of Copenhagen Author of http://www.adsl-optimizer.dk ------------------------------------------------------------------- -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, May 5, 2009 at 2:24 PM, Jesper Dangaard Brouer <hawk@diku.dk> wrote: > On Tue, 5 May 2009, David Miller wrote: > >> From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> >> Date: Tue, 5 May 2009 11:47:09 -0700 >> >>> NAK. RNBC is not a counter for buffer overruns, and so should not be >>> counted as such. >> >> I'd say technically it is, it indicates that more packets arrived than >> the available receive buffers could handle. > > I agree with DaveM. Technically it _is_ a buffer overflow, but in the host > memory not the NIC. I'm sort of pushing the system into a situation where it > cannot empty the receive buffers fast enough. > > I can fairly easily provoke this situation by adding too many iptables > rules, which (intentionally) cause high CPU load and causes ksoftirqd to run > (I'm Oprofiling netfilter modules). > > >> If anything, this is the closest this device has for this kind of >> situation, and it's useful for diagnosing problems. > > Its really useful for diagnosing problems, and I'm betting that this is a > real-life situation which people is going to experience. We might as well > help our self to more easily identify this issue when people report drop > problems. > > Notice that I'm seeing: > > rx_no_buffer_count: 136955 > rx_missed_errors: 0 > > Thus, the rx_missed_errors is zero, which according to the datasheet is the > "real" fifo drop (the MPC register, Missed Packets Count) and PCI bandwidth > problem indications. > > If we really should nitpick, then: > > adapter->net_stats.rx_missed_errors = adapter->stats.mpc > > Should then have been stored in the rx_fifo_errors. Notice that > rx_missed_errors is presented to userspace as drops (see > net/core/dev.c:2624). > > I think that both MPC and RNBC should be stored in rx_fifo_errors (and of > cause still keeping them seperate to ethtool -S). > > I'll post two patches with these changes tomorrow, for you evaluation. > > Please reconsider you NAK. > > Greetings, > Jesper Brouer > > -- the manual[1] for the hardware says: RNBC: This register counts the number of times that frames were received when there were no available buffers in host memory to store those frames (receive descriptor head and tail pointers were equal). The packet is still received if there is space in the FIFO. This register only increments if receives are enabled. This register does not increment when flow control packets are received. The critical bit "The packet is still received if there is space in the FIFO" (AND a host memory buffer becomes available) So the reason we don't want to put it in the net_stats stats for drops is that the packet *wasn't* necessarily dropped. The rx_missed errors is for packets that were definitely dropped, and is already stored in the net_stats structure.
From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> Date: Tue, 5 May 2009 14:32:04 -0700 > the manual[1] for the hardware says: > RNBC: > This register counts the number of times that frames were received > when there were no available buffers in host memory to store those > frames (receive descriptor head and tail pointers were equal). The > packet is still received if there is space in the FIFO. This register > only increments if receives are enabled. This register does not > increment when flow control packets are received. > > The critical bit "The packet is still received if there is space in > the FIFO" (AND a host memory buffer becomes available) So the reason > we don't want to put it in the net_stats stats for drops is that the > packet > *wasn't* necessarily dropped. > > The rx_missed errors is for packets that were definitely dropped, and > is already stored in the net_stats structure. While not an "rx_missed" because we do eventually take the packet, conceptually it is a "fifo overflow" in the sense that we exceeded available receive resources at the time that the packet arrived. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>Its really useful for diagnosing problems, and I'm betting >that this is a >real-life situation which people is going to experience. We >might as well >help our self to more easily identify this issue when people >report drop problems. The problem is that the RNBC aren't dropped packets as the numbers you have show. While we can agree that the MPC are the actual dropped packets and could eaily be be used in the fifo overflow count since the packets were really dropped. >I think that both MPC and RNBC should be stored in >rx_fifo_errors (and of >cause still keeping them seperate to ethtool -S). This would count RNBC packets as packets that the stack did not process, which it did. The MPC packets were not processed by the stack and should be counted as dropped. As you point out, both counts are available via ethtool -S. >I'll post two patches with these changes tomorrow, for you evaluation. Thanks, we look forward to see them. Cheers, John ----------------------------------------------------------- "...that your people will judge you on what you can build, not what you destroy.", B. Obama, 2009 -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2009-05-05 at 14:35 -0700, David Miller wrote: > From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > Date: Tue, 5 May 2009 14:32:04 -0700 > > > the manual[1] for the hardware says: > > RNBC: > > This register counts the number of times that frames were received > > when there were no available buffers in host memory to store those > > frames (receive descriptor head and tail pointers were equal). The > > packet is still received if there is space in the FIFO. This register > > only increments if receives are enabled. This register does not > > increment when flow control packets are received. > > > > The critical bit "The packet is still received if there is space in > > the FIFO" (AND a host memory buffer becomes available) So the reason > > we don't want to put it in the net_stats stats for drops is that the > > packet > > *wasn't* necessarily dropped. > > > > The rx_missed errors is for packets that were definitely dropped, and > > is already stored in the net_stats structure. > > While not an "rx_missed" because we do eventually take the > packet, conceptually it is a "fifo overflow" in the sense > that we exceeded available receive resources at the time that > the packet arrived. Yes, with this argumentation, the MPC should then be kept as "rx_missed" packets. And the RNBC stored as "rx_fifo_errors" as its an overflow indication, not a number of packets dropped.
On Wed, 6 May 2009, Jesper Dangaard Brouer wrote: > On Tue, 2009-05-05 at 14:35 -0700, David Miller wrote: > > From: Jeff Kirsher <jeffrey.t.kirsher@intel.com> > > Date: Tue, 5 May 2009 14:32:04 -0700 > > > > > the manual[1] for the hardware says: > > > RNBC: > > > This register counts the number of times that frames were received > > > when there were no available buffers in host memory to store those > > > frames (receive descriptor head and tail pointers were equal). The > > > packet is still received if there is space in the FIFO. This register > > > only increments if receives are enabled. This register does not > > > increment when flow control packets are received. > > > > > > The critical bit "The packet is still received if there is space in > > > the FIFO" (AND a host memory buffer becomes available) So the reason > > > we don't want to put it in the net_stats stats for drops is that the > > > packet > > > *wasn't* necessarily dropped. > > > > > > The rx_missed errors is for packets that were definitely dropped, and > > > is already stored in the net_stats structure. > > > > While not an "rx_missed" because we do eventually take the > > packet, conceptually it is a "fifo overflow" in the sense > > that we exceeded available receive resources at the time that > > the packet arrived. > > Yes, with this argumentation, the MPC should then be kept as "rx_missed" > packets. And the RNBC stored as "rx_fifo_errors" as its an overflow > indication, not a number of packets dropped. The way RNBC works depends on how the queues themselves are configured. Specifically, if you have packet drop enabled per queue or not will affect RNBC. In the SRRCTL registers, there is a DROP_EN bit, bit 31. If this bit is set to 1b for the queue in question, then the packet will be dropped when there are no buffers in the packet buffer. This does not mean the FIFO is full or has been overrun, it just means there's no more descriptors available in the Rx ring for that queue. In this case, RNBC is incremented, MPC is not. If bit 31 in SRRCTL is 0b, then if there's no room in the packet buffer (no more descriptors available), the device tries to store the packet in the FIFO. RNBC will *not* be incremented in this case. If there's no space in the FIFO, then the packet is dropped. RNBC still is not incremented in this case, rather MPC will be incremented, since the packet was dropped due to the FIFO being full. In 82576, according to the manual, SRRCTL bit 31 is 0b for queue 0 by default, and is 1b for all other queues by default. I hope this helps explain what the hardware is doing, and how these two counters get used in overrun cases. Cheers, -PJ Waskiewicz -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2009-05-05 at 15:38 -0700, Ronciak, John wrote: > >Its really useful for diagnosing problems, and I'm betting > >that this is a real-life situation which people is going to experience. We > >might as well help our self to more easily identify this issue when people > >report drop problems. > > The problem is that the RNBC aren't dropped packets as the numbers you > have show. Just to make it clear, I am experiencing dropped packets. The reason I positivly know this, is that I'm writing a MPEG2-TS drop detection netfilter module. Which were the only reason I discovered the packet drops and the "hidden" RNBC counter via ethtool. I have read the datasheeet, and with Jeffrey's detailed explaination, I do know that this number might be higher than the actually drops I'm experiencing. > While we can agree that the MPC are the actual dropped packets and > could eaily be be used in the fifo overflow count since the packets > were really dropped. Well, then I think we should keep MPC as drops, and use the fifo_errors as an fifo overflow indication containing RNBC count. > > I think that both MPC and RNBC should be stored in rx_fifo_errors > > (and of cause still keeping them seperate to ethtool -S). > > This would count RNBC packets as packets that the stack did not > process, which it did. The MPC packets were not processed by the > stack and should be counted as dropped. As you point out, both counts > are available via ethtool -S. > > >I'll post two patches with these changes tomorrow, for you evaluation. > > Thanks, we look forward to see them. I'll keep it to one patch (with an extra comment reflecting this disscussion), as you have convinced me that the MPC should stay as "rx_missed", as this is presented to userspace as a positive drop.
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c index 183235d..ef26e6a 100644 --- a/drivers/net/igb/igb_main.c +++ b/drivers/net/igb/igb_main.c @@ -3596,6 +3596,7 @@ void igb_update_stats(struct igb_adapter *adapter) adapter->net_stats.rx_crc_errors = adapter->stats.crcerrs; adapter->net_stats.rx_frame_errors = adapter->stats.algnerrc; adapter->net_stats.rx_missed_errors = adapter->stats.mpc; + adapter->net_stats.rx_fifo_errors = adapter->stats.rnbc; /* Tx Errors */ adapter->net_stats.tx_errors = adapter->stats.ecol +
Hardware RX fifo overruns for the 82576 is stored in the RNBC (Receive No Buffers Count) register. I choose the store the RNBC value in net_stats.rx_fifo_errors. Saving the stats in dev->net_stats makes it visible via /proc/net/dev as "fifo", and thus viewable to ifconfig as "overruns" and 'netstat -i' as "RX-OVR". The Receive No Buffers Count (RNBC) can already be queried by ethtool -S as "rx_no_buffer_count". Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk> --- drivers/net/igb/igb_main.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html