Message ID | cover.1521124830.git.pabeni@redhat.com |
---|---|
Headers | show |
Series | net:setup XPS mapping for each online CPU | expand |
On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni <pabeni@redhat.com> wrote: > Currently, most MQ netdevice setup the default XPS configuration mapping > 1-1 > the first real_num_tx_queues queues and CPUs and no mapping is created for > the CPUs with id greater then real_num_tx_queues, if any. > > As a result, the xmit path for unconnected sockets on such cores > experiences a > relevant overhead in netdev_pick_tx(), which needs to dissect each packet > and > compute its hash. > > Such scenario is easily triggered. e.g. from DNS server under relevant > load, as > the user-space process is moved away from the CPUs serving the softirqs > (note: > this is beneficial for the overall DNS server performances). > > This series introduces an helper to easily setup up XPS mapping for all the > online CPUs, and use it in the ixgbe driver, demonstrating a relevant > performance improvement in the above scenario. > > I truly believe XPS should not be setup by devices. XPS is policy, and policy does belong to user space. User space knows that CPU 2 and 3 (pure examples) are reserved for storage interrupts, not NIC ones. Note that if XPS is not setup, MQ queue selection is just fine by default ;) <div dir="ltr"><br><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni <<a href="mailto:pabeni@redhat.com">pabeni@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Currently, most MQ netdevice setup the default XPS configuration mapping 1-1<br> the first real_num_tx_queues queues and CPUs and no mapping is created for<br> the CPUs with id greater then real_num_tx_queues, if any.<br> <br> As a result, the xmit path for unconnected sockets on such cores experiences a<br> relevant overhead in netdev_pick_tx(), which needs to dissect each packet and<br> compute its hash.<br> <br> Such scenario is easily triggered. e.g. from DNS server under relevant load, as<br> the user-space process is moved away from the CPUs serving the softirqs (note:<br> this is beneficial for the overall DNS server performances).<br> <br> This series introduces an helper to easily setup up XPS mapping for all the<br> online CPUs, and use it in the ixgbe driver, demonstrating a relevant<br> performance improvement in the above scenario.<br><br></blockquote><div><br></div><div>I truly believe XPS should not be setup by devices.</div><div><br></div><div>XPS is policy, and policy does belong to user space.</div><div><br></div><div>User space knows that CPU 2 and 3 (pure examples) are reserved for storage interrupts, not NIC ones.</div><div><br></div><div>Note that if XPS is not setup, MQ queue selection is just fine by default ;)</div><div><br></div><div> </div></div></div>
On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni <pabeni@redhat.com> wrote: > Currently, most MQ netdevice setup the default XPS configuration mapping 1-1 > the first real_num_tx_queues queues and CPUs and no mapping is created for > the CPUs with id greater then real_num_tx_queues, if any. > As a result, the xmit path for unconnected sockets on such cores experiences a > relevant overhead in netdev_pick_tx(), which needs to dissect each packet and > compute its hash. > Such scenario is easily triggered. e.g. from DNS server under relevant load, as > the user-space process is moved away from the CPUs serving the softirqs (note: > this is beneficial for the overall DNS server performances). > This series introduces an helper to easily setup up XPS mapping for all the > online CPUs, and use it in the ixgbe driver, demonstrating a relevant > performance improvement in the above scenario. > Paolo Abeni (2): > net: introduce netif_set_xps() > ixgbe: setup XPS via netif_set_xps() Resent, not HTML this time, sorry for duplication. I truly believe XPS should not be setup by devices. XPS is policy, and policy does belong to user space. User space knows that CPU 2 and 3 (pure examples) are reserved for storage interrupts, not NIC ones. Note that if XPS is not setup, MQ queue selection is just fine by default ;
Hi, On Thu, 2018-03-15 at 15:31 +0000, Eric Dumazet wrote: > On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni <pabeni@redhat.com> wrote: > > > Currently, most MQ netdevice setup the default XPS configuration mapping > > 1-1 > > the first real_num_tx_queues queues and CPUs and no mapping is created for > > the CPUs with id greater then real_num_tx_queues, if any. > > As a result, the xmit path for unconnected sockets on such cores > > experiences a > > relevant overhead in netdev_pick_tx(), which needs to dissect each packet > > and > > compute its hash. > > Such scenario is easily triggered. e.g. from DNS server under relevant > > load, as > > the user-space process is moved away from the CPUs serving the softirqs > > (note: > > this is beneficial for the overall DNS server performances). > > This series introduces an helper to easily setup up XPS mapping for all > > the > > online CPUs, and use it in the ixgbe driver, demonstrating a relevant > > performance improvement in the above scenario. > > Paolo Abeni (2): > > net: introduce netif_set_xps() > > ixgbe: setup XPS via netif_set_xps() > > > Resent, not HTML this time, sorry for duplication. > > I truly believe XPS should not be setup by devices. > > XPS is policy, and policy does belong to user space. Thank you for your comments! As general principle, I agree policies should be in user-space, but I also think that the kernel should provide a reasonable default. Many MQ devices already configure XPS and their default is AFAICS sub-optimal. > Note that if XPS is not setup, MQ queue selection is just fine by default ; I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect() overhead in xmit path. Cheers, Paolo
On Thu, Mar 15, 2018 at 8:51 AM Paolo Abeni <pabeni@redhat.com> wrote: > Hi, > On Thu, 2018-03-15 at 15:31 +0000, Eric Dumazet wrote: > > On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni <pabeni@redhat.com> wrote: > > > > > Currently, most MQ netdevice setup the default XPS configuration mapping > > > > 1-1 > > > the first real_num_tx_queues queues and CPUs and no mapping is created for > > > the CPUs with id greater then real_num_tx_queues, if any. > > > As a result, the xmit path for unconnected sockets on such cores > > > > experiences a > > > relevant overhead in netdev_pick_tx(), which needs to dissect each packet > > > > and > > > compute its hash. > > > Such scenario is easily triggered. e.g. from DNS server under relevant > > > > load, as > > > the user-space process is moved away from the CPUs serving the softirqs > > > > (note: > > > this is beneficial for the overall DNS server performances). > > > This series introduces an helper to easily setup up XPS mapping for all > > > > the > > > online CPUs, and use it in the ixgbe driver, demonstrating a relevant > > > performance improvement in the above scenario. > > > Paolo Abeni (2): > > > net: introduce netif_set_xps() > > > ixgbe: setup XPS via netif_set_xps() > > > > > > Resent, not HTML this time, sorry for duplication. > > > > I truly believe XPS should not be setup by devices. > > > > XPS is policy, and policy does belong to user space. > Thank you for your comments! > As general principle, I agree policies should be in user-space, but I > also think that the kernel should provide a reasonable default. Many MQ > devices already configure XPS and their default is AFAICS sub-optimal. > > Note that if XPS is not setup, MQ queue selection is just fine by default ; > I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS > we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect() > overhead in xmit path. Then fix this if you want, instead of fixing one NIC only, or by enforcing XPS by all NIC. For unconnected sockets, picking the TX queue based on current cpu is good, we do not have to enforce ordering as much as possible. (pfifo_fast no longer can enforce it anyway)
Hi, On Thu, 2018-03-15 at 15:59 +0000, Eric Dumazet wrote: > On Thu, Mar 15, 2018 at 8:51 AM Paolo Abeni <pabeni@redhat.com> wrote: > > I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS > > we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect() > > overhead in xmit path. > > Then fix this if you want, instead of fixing one NIC only, or by enforcing > XPS by all NIC. > > For unconnected sockets, picking the TX queue based on current cpu is good, > we do not have to enforce ordering as much as possible. > > (pfifo_fast no longer can enforce it anyway) Thank you for the prompt reply. I'm double checking to avoid misinterpretation on my side: are you suggesting to plug a CPU-based selection logic for unconnected sockets in netdev_pick_tx() or to cook patches like 2/2 for all the relevant NICs? Thanks! Paolo
On Thu, Mar 15, 2018 at 10:20 AM, Paolo Abeni <pabeni@redhat.com> wrote: > Hi, > > On Thu, 2018-03-15 at 15:59 +0000, Eric Dumazet wrote: >> On Thu, Mar 15, 2018 at 8:51 AM Paolo Abeni <pabeni@redhat.com> wrote: >> > I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS >> > we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect() >> > overhead in xmit path. >> >> Then fix this if you want, instead of fixing one NIC only, or by enforcing >> XPS by all NIC. >> >> For unconnected sockets, picking the TX queue based on current cpu is good, >> we do not have to enforce ordering as much as possible. >> >> (pfifo_fast no longer can enforce it anyway) > > Thank you for the prompt reply. > > I'm double checking to avoid misinterpretation on my side: are you > suggesting to plug a CPU-based selection logic for unconnected sockets > in netdev_pick_tx() or to cook patches like 2/2 for all the relevant > NICs? > > Thanks! > > Paolo We just need to watch out for any possible side effects. For example using XPS on a virtualization host has been problematic as you end up with the traffic getting reordered if the VM jumps from CPU to CPU. Thanks. - Alex