| Message ID | CAB1FvaJr4GLbQ1wjzev1sjh78gscNEyZ9EEWk1yq+AedtssySA@mail.gmail.com |
|---|---|
| State | Accepted |
| Headers | show |
On Mon, Aug 29, 2016 at 03:32:41PM -0700, Neil McKee wrote: > This patch changes the order of the steps that are followed > every second in the sFlow agent. By moving the receiver_tick() > step to the end, we ensure that any counters that were polled > during the poller_tick() step are flushed immediately to the > sFlow collector. This eliminates what was a variable time-delay > between counters being polled and being flushed. > > The variable time-delay that this eliminates could be up to > a second because counters lingering in the output buffer could be > flushed at any time by the arrival of random packet-samples. > > Since the sFlow standard does not require that a poll-timestamp be sent > along with the counters the collector must use his receive-time as the > timestamp, so that extra second of variable delay was "stretching or > shrinking" the time between successive counter readings. This > affected any counter-rate calculation that was based only on the delta > between sucessive samples. The effect was small with a polling > interval of 60 seconds: just +/- 2%. But the effect grew larger > when faster polling was configured. For example, if the counters > were pushed every 5 seconds then the instantaneous rate > calculations could wander by +/- 20%. For a thorough analysis > of this problem, see Rick Jones' paper: > > "High Frequency sFlow v5 Counter Sampling" > ftp://ftp.netperf.org/papers/high_freq_sflow/hf_sflow_counters.pdf > > So this patch makes it possible to obtain usable results even > when high-frequency polling is configured. > > Signed-off-by: Neil McKee <neil.mckee@inmon.com> Thanks, applied to master and branch-2.6.
This one perturbs the output ordering just enough to fail two of the sflow unit tests. Sorry I didn't notice that before. I'll post another patch for that. Neil ------ Neil McKee InMon Corp. http://www.inmon.com On Fri, Sep 2, 2016 at 11:47 AM, Ben Pfaff <blp@ovn.org> wrote: > On Mon, Aug 29, 2016 at 03:32:41PM -0700, Neil McKee wrote: >> This patch changes the order of the steps that are followed >> every second in the sFlow agent. By moving the receiver_tick() >> step to the end, we ensure that any counters that were polled >> during the poller_tick() step are flushed immediately to the >> sFlow collector. This eliminates what was a variable time-delay >> between counters being polled and being flushed. >> >> The variable time-delay that this eliminates could be up to >> a second because counters lingering in the output buffer could be >> flushed at any time by the arrival of random packet-samples. >> >> Since the sFlow standard does not require that a poll-timestamp be sent >> along with the counters the collector must use his receive-time as the >> timestamp, so that extra second of variable delay was "stretching or >> shrinking" the time between successive counter readings. This >> affected any counter-rate calculation that was based only on the delta >> between sucessive samples. The effect was small with a polling >> interval of 60 seconds: just +/- 2%. But the effect grew larger >> when faster polling was configured. For example, if the counters >> were pushed every 5 seconds then the instantaneous rate >> calculations could wander by +/- 20%. For a thorough analysis >> of this problem, see Rick Jones' paper: >> >> "High Frequency sFlow v5 Counter Sampling" >> ftp://ftp.netperf.org/papers/high_freq_sflow/hf_sflow_counters.pdf >> >> So this patch makes it possible to obtain usable results even >> when high-frequency polling is configured. >> >> Signed-off-by: Neil McKee <neil.mckee@inmon.com> > > Thanks, applied to master and branch-2.6.
I see the failing tests. If you can read the new results of the tests and verify for me that they're still a correct set of results, then I'll push a fix that updates the expected results. On Fri, Sep 02, 2016 at 12:03:21PM -0700, Neil McKee wrote: > This one perturbs the output ordering just enough to fail two of the > sflow unit tests. Sorry I didn't notice that before. I'll post > another patch for that. > > Neil > ------ > Neil McKee > InMon Corp. > http://www.inmon.com > > > On Fri, Sep 2, 2016 at 11:47 AM, Ben Pfaff <blp@ovn.org> wrote: > > On Mon, Aug 29, 2016 at 03:32:41PM -0700, Neil McKee wrote: > >> This patch changes the order of the steps that are followed > >> every second in the sFlow agent. By moving the receiver_tick() > >> step to the end, we ensure that any counters that were polled > >> during the poller_tick() step are flushed immediately to the > >> sFlow collector. This eliminates what was a variable time-delay > >> between counters being polled and being flushed. > >> > >> The variable time-delay that this eliminates could be up to > >> a second because counters lingering in the output buffer could be > >> flushed at any time by the arrival of random packet-samples. > >> > >> Since the sFlow standard does not require that a poll-timestamp be sent > >> along with the counters the collector must use his receive-time as the > >> timestamp, so that extra second of variable delay was "stretching or > >> shrinking" the time between successive counter readings. This > >> affected any counter-rate calculation that was based only on the delta > >> between sucessive samples. The effect was small with a polling > >> interval of 60 seconds: just +/- 2%. But the effect grew larger > >> when faster polling was configured. For example, if the counters > >> were pushed every 5 seconds then the instantaneous rate > >> calculations could wander by +/- 20%. For a thorough analysis > >> of this problem, see Rick Jones' paper: > >> > >> "High Frequency sFlow v5 Counter Sampling" > >> ftp://ftp.netperf.org/papers/high_freq_sflow/hf_sflow_counters.pdf > >> > >> So this patch makes it possible to obtain usable results even > >> when high-frequency polling is configured. > >> > >> Signed-off-by: Neil McKee <neil.mckee@inmon.com> > > > > Thanks, applied to master and branch-2.6.
diff --git a/lib/sflow_agent.c b/lib/sflow_agent.c index 9c2e028..878c3da 100644 --- a/lib/sflow_agent.c +++ b/lib/sflow_agent.c @@ -128,12 +128,15 @@ void sfl_agent_tick(SFLAgent *agent, time_t now) SFLSampler *sm = agent->samplers; SFLPoller *pl = agent->pollers; agent->now = now; - /* receivers use ticks to flush send data */ - for(; rcv != NULL; rcv = rcv->nxt) sfl_receiver_tick(rcv, now); /* samplers use ticks to decide when they are sampling too fast */ for(; sm != NULL; sm = sm->nxt) sfl_sampler_tick(sm, now); /* pollers use ticks to decide when to ask for counters */ for(; pl != NULL; pl = pl->nxt) sfl_poller_tick(pl, now); + /* receivers use ticks to flush send data. By doing this + * step last we ensure that fresh counters polled during + * sfl_poller_tick() above will be flushed promptly. + */ + for(; rcv != NULL; rcv = rcv->nxt) sfl_receiver_tick(rcv, now); } /*_________________---------------------------__________________
This patch changes the order of the steps that are followed every second in the sFlow agent. By moving the receiver_tick() step to the end, we ensure that any counters that were polled during the poller_tick() step are flushed immediately to the sFlow collector. This eliminates what was a variable time-delay between counters being polled and being flushed. The variable time-delay that this eliminates could be up to a second because counters lingering in the output buffer could be flushed at any time by the arrival of random packet-samples. Since the sFlow standard does not require that a poll-timestamp be sent along with the counters the collector must use his receive-time as the timestamp, so that extra second of variable delay was "stretching or shrinking" the time between successive counter readings. This affected any counter-rate calculation that was based only on the delta between sucessive samples. The effect was small with a polling interval of 60 seconds: just +/- 2%. But the effect grew larger when faster polling was configured. For example, if the counters were pushed every 5 seconds then the instantaneous rate calculations could wander by +/- 20%. For a thorough analysis of this problem, see Rick Jones' paper: "High Frequency sFlow v5 Counter Sampling" ftp://ftp.netperf.org/papers/high_freq_sflow/hf_sflow_counters.pdf So this patch makes it possible to obtain usable results even when high-frequency polling is configured. Signed-off-by: Neil McKee <neil.mckee@inmon.com> --- lib/sflow_agent.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)