Message ID | 20180419125729.15198-3-stephen@that.guru |
---|---|
State | Accepted |
Delegated to: | Ian Stokes |
Headers | show |
Series | Split up the DPDK how-to | expand |
On 04/19/2018 01:57 PM, Stephen Finucane wrote: > This continues the breakup of the huge DPDK "howto" into smaller > components. There are a couple of related changes included, such as > using "Rx queue" instead of "rxq" and noting how Tx queues cannot be > configured. > > Signed-off-by: Stephen Finucane <stephen@that.guru> > --- > v3: > - Add new files to automake.mk > - Add additional 'versionchanged' information for 'pmd-rxq-show' command > v2: > - Add cross-references from 'pmd' doc to 'vhost-user' and 'phy' docs > - Add 'versionchanged' warning about automatic assignment of Rx queues > - Add a 'todo' to describe Tx queue behavior > --- > Documentation/automake.mk | 1 + > Documentation/howto/dpdk.rst | 86 ----------------- > Documentation/topics/dpdk/index.rst | 1 + > Documentation/topics/dpdk/phy.rst | 12 +++ > Documentation/topics/dpdk/pmd.rst | 161 +++++++++++++++++++++++++++++++ > Documentation/topics/dpdk/vhost-user.rst | 17 ++-- > 6 files changed, 183 insertions(+), 95 deletions(-) > create mode 100644 Documentation/topics/dpdk/pmd.rst > > diff --git a/Documentation/automake.mk b/Documentation/automake.mk > index cd8f3dc0d..878de4349 100644 > --- a/Documentation/automake.mk > +++ b/Documentation/automake.mk > @@ -35,6 +35,7 @@ DOC_SOURCE = \ > Documentation/topics/design.rst \ > Documentation/topics/dpdk/index.rst \ > Documentation/topics/dpdk/phy.rst \ > + Documentation/topics/dpdk/pmd.rst \ > Documentation/topics/dpdk/ring.rst \ > Documentation/topics/dpdk/vhost-user.rst \ > Documentation/topics/testing.rst \ > diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst > index 79b626c76..388728363 100644 > --- a/Documentation/howto/dpdk.rst > +++ b/Documentation/howto/dpdk.rst > @@ -81,92 +81,6 @@ To stop ovs-vswitchd & delete bridge, run:: > $ ovs-appctl -t ovsdb-server exit > $ ovs-vsctl del-br br0 > > -PMD Thread Statistics > ---------------------- > - > -To show current stats:: > - > - $ ovs-appctl dpif-netdev/pmd-stats-show > - > -To clear previous stats:: > - > - $ ovs-appctl dpif-netdev/pmd-stats-clear > - > -Port/RXQ Assigment to PMD Threads > ---------------------------------- > - > -To show port/rxq assignment:: > - > - $ ovs-appctl dpif-netdev/pmd-rxq-show > - > -To change default rxq assignment to pmd threads, rxqs may be manually pinned to > -desired cores using:: > - > - $ ovs-vsctl set Interface <iface> \ > - other_config:pmd-rxq-affinity=<rxq-affinity-list> > - > -where: > - > -- ``<rxq-affinity-list>`` is a CSV list of ``<queue-id>:<core-id>`` values > - > -For example:: > - > - $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 \ > - other_config:pmd-rxq-affinity="0:3,1:7,3:8" > - > -This will ensure: > - > -- Queue #0 pinned to core 3 > -- Queue #1 pinned to core 7 > -- Queue #2 not pinned > -- Queue #3 pinned to core 8 > - > -After that PMD threads on cores where RX queues was pinned will become > -``isolated``. This means that this thread will poll only pinned RX queues. > - > -.. warning:: > - If there are no ``non-isolated`` PMD threads, ``non-pinned`` RX queues will > - not be polled. Also, if provided ``core_id`` is not available (ex. this > - ``core_id`` not in ``pmd-cpu-mask``), RX queue will not be polled by any PMD > - thread. > - > -If pmd-rxq-affinity is not set for rxqs, they will be assigned to pmds (cores) > -automatically. The processing cycles that have been stored for each rxq > -will be used where known to assign rxqs to pmd based on a round robin of the > -sorted rxqs. > - > -For example, in the case where here there are 5 rxqs and 3 cores (e.g. 3,7,8) > -available, and the measured usage of core cycles per rxq over the last > -interval is seen to be: > - > -- Queue #0: 30% > -- Queue #1: 80% > -- Queue #3: 60% > -- Queue #4: 70% > -- Queue #5: 10% > - > -The rxqs will be assigned to cores 3,7,8 in the following order: > - > -Core 3: Q1 (80%) | > -Core 7: Q4 (70%) | Q5 (10%) > -core 8: Q3 (60%) | Q0 (30%) > - > -To see the current measured usage history of pmd core cycles for each rxq:: > - > - $ ovs-appctl dpif-netdev/pmd-rxq-show > - > -.. note:: > - > - A history of one minute is recorded and shown for each rxq to allow for > - traffic pattern spikes. An rxq's pmd core cycles usage changes due to traffic > - pattern or reconfig changes will take one minute before they are fully > - reflected in the stats. > - > -Rxq to pmds assignment takes place whenever there are configuration changes > -or can be triggered by using:: > - > - $ ovs-appctl dpif-netdev/pmd-rxq-rebalance > - > QoS > --- > > diff --git a/Documentation/topics/dpdk/index.rst b/Documentation/topics/dpdk/index.rst > index 5f836a6e9..dfde88377 100644 > --- a/Documentation/topics/dpdk/index.rst > +++ b/Documentation/topics/dpdk/index.rst > @@ -31,3 +31,4 @@ The DPDK Datapath > phy > vhost-user > ring > + pmd > diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst > index a3f8b475c..ad191dad0 100644 > --- a/Documentation/topics/dpdk/phy.rst > +++ b/Documentation/topics/dpdk/phy.rst > @@ -113,3 +113,15 @@ tool:: > For more information, refer to the `DPDK documentation <dpdk-drivers>`__. > > .. _dpdk-drivers: http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html > + > +.. _dpdk-phy-multiqueue: > + > +Multiqueue > +---------- > + > +Poll Mode Driver (PMD) threads are the threads that do the heavy lifting for > +the DPDK datapath. Correct configuration of PMD threads and the Rx queues they > +utilize is a requirement in order to deliver the high-performance possible with > +DPDK acceleration. It is possible to configure multiple Rx queues for ``dpdk`` > +ports, thus ensuring this is not a bottleneck for performance. For information > +on configuring PMD threads, refer to :doc:`pmd`. > diff --git a/Documentation/topics/dpdk/pmd.rst b/Documentation/topics/dpdk/pmd.rst > new file mode 100644 > index 000000000..a652720e2 > --- /dev/null > +++ b/Documentation/topics/dpdk/pmd.rst > @@ -0,0 +1,161 @@ > +.. > + Licensed under the Apache License, Version 2.0 (the "License"); you may > + not use this file except in compliance with the License. You may obtain > + a copy of the License at > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > + Unless required by applicable law or agreed to in writing, software > + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT > + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the > + License for the specific language governing permissions and limitations > + under the License. > + > + Convention for heading levels in Open vSwitch documentation: > + > + ======= Heading 0 (reserved for the title in a document) > + ------- Heading 1 > + ~~~~~~~ Heading 2 > + +++++++ Heading 3 > + ''''''' Heading 4 > + > + Avoid deeper levels because they do not render well. > + > +=========== > +PMD Threads > +=========== > + > +Poll Mode Driver (PMD) threads are the threads that do the heavy lifting for > +the DPDK datapath and perform tasks such as continuous polling of input ports > +for packets, classifying packets once received, and executing actions on the > +packets once they are classified. > + > +PMD threads utilize Receive (Rx) and Transmit (Tx) queues, commonly known as > +*rxq*\s and *txq*\s. While Tx queue configuration happens automatically, Rx > +queues can be configured by the user. This can happen in one of two ways: > + > +- For physical interfaces, configuration is done using the > + :program:`ovs-appctl` utility. > + > +- For virtual interfaces, configuration is done using the :program:`ovs-appctl` > + utility, but this configuration must be reflected in the guest configuration > + (e.g. QEMU command line arguments). > + > +The :program:`ovs-appctl` utility also provides a number of commands for > +querying PMD threads and their respective queues. This, and all of the above, > +is discussed here. > + > +.. todo:: > + > + Add an overview of Tx queues including numbers created, how they relate to > + PMD threads, etc. > + > +PMD Thread Statistics > +--------------------- > + > +To show current stats:: > + > + $ ovs-appctl dpif-netdev/pmd-stats-show > + > +To clear previous stats:: > + > + $ ovs-appctl dpif-netdev/pmd-stats-clear > + > +Port/Rx Queue Assigment to PMD Threads > +-------------------------------------- > + > +.. todo:: > + > + This needs a more detailed overview of *why* this should be done, along with > + the impact on things like NUMA affinity. > + > +Correct configuration of PMD threads and the Rx queues they utilize is a > +requirement in order to achieve maximum performance. This is particularly true > +for enabling things like multiqueue for :ref:`physical <dpdk-phy-multiqueue>` > +and :ref:`vhost-user <dpdk-vhost-user>` interfaces. > + > +To show port/Rx queue assignment:: > + > + $ ovs-appctl dpif-netdev/pmd-rxq-show > + > +Rx queues may be manually pinned to cores. This will change the default Rx > +queue assignment to PMD threads:: > + > + $ ovs-vsctl set Interface <iface> \ > + other_config:pmd-rxq-affinity=<rxq-affinity-list> > + > +where: > + > +- ``<rxq-affinity-list>`` is a CSV list of ``<queue-id>:<core-id>`` values > + > +For example:: > + > + $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 \ > + other_config:pmd-rxq-affinity="0:3,1:7,3:8" > + > +This will ensure there are *4* Rx queues and that these queues are configured > +like so: > + > +- Queue #0 pinned to core 3 > +- Queue #1 pinned to core 7 > +- Queue #2 not pinned > +- Queue #3 pinned to core 8 > + > +PMD threads on cores where Rx queues are *pinned* will become *isolated*. This > +means that this thread will only poll the *pinned* Rx queues. > + > +.. warning:: > + > + If there are no *non-isolated* PMD threads, *non-pinned* RX queues will not > + be polled. Also, if the provided ``<core-id>`` is not available (e.g. the > + ``<core-id>`` is not in ``pmd-cpu-mask``), the RX queue will not be polled > + by any PMD thread. > + > +If ``pmd-rxq-affinity`` is not set for Rx queues, they will be assigned to PMDs > +(cores) automatically. Where known, the processing cycles that have been stored > +for each Rx queue will be used to assign Rx queue to PMDs based on a round > +robin of the sorted Rx queues. For example, take the following example, where > +there are five Rx queues and three cores - 3, 7, and 8 - available and the > +measured usage of core cycles per Rx queue over the last interval is seen to > +be: > + > +- Queue #0: 30% > +- Queue #1: 80% > +- Queue #3: 60% > +- Queue #4: 70% > +- Queue #5: 10% > + > +The Rx queues will be assigned to the cores in the following order:: > + > + Core 3: Q1 (80%) | > + Core 7: Q4 (70%) | Q5 (10%) > + Core 8: Q3 (60%) | Q0 (30%) > + > +To see the current measured usage history of PMD core cycles for each Rx > +queue:: > + > + $ ovs-appctl dpif-netdev/pmd-rxq-show > + > +.. note:: > + > + A history of one minute is recorded and shown for each Rx queue to allow for > + traffic pattern spikes. Any changes in the Rx queue's PMD core cycles usage, > + due to traffic pattern or reconfig changes, will take one minute to be fully > + reflected in the stats. > + > +Rx queue to PMD assignment takes place whenever there are configuration changes > +or can be triggered by using:: > + > + $ ovs-appctl dpif-netdev/pmd-rxq-rebalance > + > +.. versionchanged:: 2.8.0 > + > + Automatic assignment of Rx queues to PMDs and the two related commands, > + ``pmd-rxq-show`` and ``pmd-rxq-rebalance``, were added in OVS 2.8.0. Prior pmd-rxq-show added in OVS 2.6, modified in 2.9 as noted below pmd-rxq-rebalance added in OVS 2.9 > + to this, behavior was round-robin and processing cycles were not taken into > + consideration. Tracking for stats was not available. > + > +.. versionchanged:: 2.9.0 > + > + The output of ``pmd-rxq-show`` was modified to include utilization as a > + percentage. > diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst > index ca8a3289f..6f794f296 100644 > --- a/Documentation/topics/dpdk/vhost-user.rst > +++ b/Documentation/topics/dpdk/vhost-user.rst > @@ -130,11 +130,10 @@ an additional set of parameters:: > -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce > -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 > > -In addition, QEMU must allocate the VM's memory on hugetlbfs. vhost-user > -ports access a virtio-net device's virtual rings and packet buffers mapping the > -VM's physical memory on hugetlbfs. To enable vhost-user ports to map the VM's > -memory into their process address space, pass the following parameters to > -QEMU:: > +In addition, QEMU must allocate the VM's memory on hugetlbfs. vhost-user ports > +access a virtio-net device's virtual rings and packet buffers mapping the VM's > +physical memory on hugetlbfs. To enable vhost-user ports to map the VM's memory > +into their process address space, pass the following parameters to QEMU:: > > -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on > -numa node,memdev=mem -mem-prealloc > @@ -154,18 +153,18 @@ where: > The number of vectors, which is ``$q`` * 2 + 2 > > The vhost-user interface will be automatically reconfigured with required > -number of rx and tx queues after connection of virtio device. Manual > +number of Rx and Tx queues after connection of virtio device. Manual > configuration of ``n_rxq`` is not supported because OVS will work properly only > if ``n_rxq`` will match number of queues configured in QEMU. > > -A least 2 PMDs should be configured for the vswitch when using multiqueue. > +A least two PMDs should be configured for the vswitch when using multiqueue. > Using a single PMD will cause traffic to be enqueued to the same vhost queue > rather than being distributed among different vhost queues for a vhost-user > interface. > > If traffic destined for a VM configured with multiqueue arrives to the vswitch > -via a physical DPDK port, then the number of rxqs should also be set to at > -least 2 for that physical DPDK port. This is required to increase the > +via a physical DPDK port, then the number of Rx queues should also be set to at > +least two for that physical DPDK port. This is required to increase the > probability that a different PMD will handle the multiqueue transmission to the > guest using a different vhost queue. > >
diff --git a/Documentation/automake.mk b/Documentation/automake.mk index cd8f3dc0d..878de4349 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -35,6 +35,7 @@ DOC_SOURCE = \ Documentation/topics/design.rst \ Documentation/topics/dpdk/index.rst \ Documentation/topics/dpdk/phy.rst \ + Documentation/topics/dpdk/pmd.rst \ Documentation/topics/dpdk/ring.rst \ Documentation/topics/dpdk/vhost-user.rst \ Documentation/topics/testing.rst \ diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index 79b626c76..388728363 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -81,92 +81,6 @@ To stop ovs-vswitchd & delete bridge, run:: $ ovs-appctl -t ovsdb-server exit $ ovs-vsctl del-br br0 -PMD Thread Statistics ---------------------- - -To show current stats:: - - $ ovs-appctl dpif-netdev/pmd-stats-show - -To clear previous stats:: - - $ ovs-appctl dpif-netdev/pmd-stats-clear - -Port/RXQ Assigment to PMD Threads ---------------------------------- - -To show port/rxq assignment:: - - $ ovs-appctl dpif-netdev/pmd-rxq-show - -To change default rxq assignment to pmd threads, rxqs may be manually pinned to -desired cores using:: - - $ ovs-vsctl set Interface <iface> \ - other_config:pmd-rxq-affinity=<rxq-affinity-list> - -where: - -- ``<rxq-affinity-list>`` is a CSV list of ``<queue-id>:<core-id>`` values - -For example:: - - $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 \ - other_config:pmd-rxq-affinity="0:3,1:7,3:8" - -This will ensure: - -- Queue #0 pinned to core 3 -- Queue #1 pinned to core 7 -- Queue #2 not pinned -- Queue #3 pinned to core 8 - -After that PMD threads on cores where RX queues was pinned will become -``isolated``. This means that this thread will poll only pinned RX queues. - -.. warning:: - If there are no ``non-isolated`` PMD threads, ``non-pinned`` RX queues will - not be polled. Also, if provided ``core_id`` is not available (ex. this - ``core_id`` not in ``pmd-cpu-mask``), RX queue will not be polled by any PMD - thread. - -If pmd-rxq-affinity is not set for rxqs, they will be assigned to pmds (cores) -automatically. The processing cycles that have been stored for each rxq -will be used where known to assign rxqs to pmd based on a round robin of the -sorted rxqs. - -For example, in the case where here there are 5 rxqs and 3 cores (e.g. 3,7,8) -available, and the measured usage of core cycles per rxq over the last -interval is seen to be: - -- Queue #0: 30% -- Queue #1: 80% -- Queue #3: 60% -- Queue #4: 70% -- Queue #5: 10% - -The rxqs will be assigned to cores 3,7,8 in the following order: - -Core 3: Q1 (80%) | -Core 7: Q4 (70%) | Q5 (10%) -core 8: Q3 (60%) | Q0 (30%) - -To see the current measured usage history of pmd core cycles for each rxq:: - - $ ovs-appctl dpif-netdev/pmd-rxq-show - -.. note:: - - A history of one minute is recorded and shown for each rxq to allow for - traffic pattern spikes. An rxq's pmd core cycles usage changes due to traffic - pattern or reconfig changes will take one minute before they are fully - reflected in the stats. - -Rxq to pmds assignment takes place whenever there are configuration changes -or can be triggered by using:: - - $ ovs-appctl dpif-netdev/pmd-rxq-rebalance - QoS --- diff --git a/Documentation/topics/dpdk/index.rst b/Documentation/topics/dpdk/index.rst index 5f836a6e9..dfde88377 100644 --- a/Documentation/topics/dpdk/index.rst +++ b/Documentation/topics/dpdk/index.rst @@ -31,3 +31,4 @@ The DPDK Datapath phy vhost-user ring + pmd diff --git a/Documentation/topics/dpdk/phy.rst b/Documentation/topics/dpdk/phy.rst index a3f8b475c..ad191dad0 100644 --- a/Documentation/topics/dpdk/phy.rst +++ b/Documentation/topics/dpdk/phy.rst @@ -113,3 +113,15 @@ tool:: For more information, refer to the `DPDK documentation <dpdk-drivers>`__. .. _dpdk-drivers: http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html + +.. _dpdk-phy-multiqueue: + +Multiqueue +---------- + +Poll Mode Driver (PMD) threads are the threads that do the heavy lifting for +the DPDK datapath. Correct configuration of PMD threads and the Rx queues they +utilize is a requirement in order to deliver the high-performance possible with +DPDK acceleration. It is possible to configure multiple Rx queues for ``dpdk`` +ports, thus ensuring this is not a bottleneck for performance. For information +on configuring PMD threads, refer to :doc:`pmd`. diff --git a/Documentation/topics/dpdk/pmd.rst b/Documentation/topics/dpdk/pmd.rst new file mode 100644 index 000000000..a652720e2 --- /dev/null +++ b/Documentation/topics/dpdk/pmd.rst @@ -0,0 +1,161 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +=========== +PMD Threads +=========== + +Poll Mode Driver (PMD) threads are the threads that do the heavy lifting for +the DPDK datapath and perform tasks such as continuous polling of input ports +for packets, classifying packets once received, and executing actions on the +packets once they are classified. + +PMD threads utilize Receive (Rx) and Transmit (Tx) queues, commonly known as +*rxq*\s and *txq*\s. While Tx queue configuration happens automatically, Rx +queues can be configured by the user. This can happen in one of two ways: + +- For physical interfaces, configuration is done using the + :program:`ovs-appctl` utility. + +- For virtual interfaces, configuration is done using the :program:`ovs-appctl` + utility, but this configuration must be reflected in the guest configuration + (e.g. QEMU command line arguments). + +The :program:`ovs-appctl` utility also provides a number of commands for +querying PMD threads and their respective queues. This, and all of the above, +is discussed here. + +.. todo:: + + Add an overview of Tx queues including numbers created, how they relate to + PMD threads, etc. + +PMD Thread Statistics +--------------------- + +To show current stats:: + + $ ovs-appctl dpif-netdev/pmd-stats-show + +To clear previous stats:: + + $ ovs-appctl dpif-netdev/pmd-stats-clear + +Port/Rx Queue Assigment to PMD Threads +-------------------------------------- + +.. todo:: + + This needs a more detailed overview of *why* this should be done, along with + the impact on things like NUMA affinity. + +Correct configuration of PMD threads and the Rx queues they utilize is a +requirement in order to achieve maximum performance. This is particularly true +for enabling things like multiqueue for :ref:`physical <dpdk-phy-multiqueue>` +and :ref:`vhost-user <dpdk-vhost-user>` interfaces. + +To show port/Rx queue assignment:: + + $ ovs-appctl dpif-netdev/pmd-rxq-show + +Rx queues may be manually pinned to cores. This will change the default Rx +queue assignment to PMD threads:: + + $ ovs-vsctl set Interface <iface> \ + other_config:pmd-rxq-affinity=<rxq-affinity-list> + +where: + +- ``<rxq-affinity-list>`` is a CSV list of ``<queue-id>:<core-id>`` values + +For example:: + + $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 \ + other_config:pmd-rxq-affinity="0:3,1:7,3:8" + +This will ensure there are *4* Rx queues and that these queues are configured +like so: + +- Queue #0 pinned to core 3 +- Queue #1 pinned to core 7 +- Queue #2 not pinned +- Queue #3 pinned to core 8 + +PMD threads on cores where Rx queues are *pinned* will become *isolated*. This +means that this thread will only poll the *pinned* Rx queues. + +.. warning:: + + If there are no *non-isolated* PMD threads, *non-pinned* RX queues will not + be polled. Also, if the provided ``<core-id>`` is not available (e.g. the + ``<core-id>`` is not in ``pmd-cpu-mask``), the RX queue will not be polled + by any PMD thread. + +If ``pmd-rxq-affinity`` is not set for Rx queues, they will be assigned to PMDs +(cores) automatically. Where known, the processing cycles that have been stored +for each Rx queue will be used to assign Rx queue to PMDs based on a round +robin of the sorted Rx queues. For example, take the following example, where +there are five Rx queues and three cores - 3, 7, and 8 - available and the +measured usage of core cycles per Rx queue over the last interval is seen to +be: + +- Queue #0: 30% +- Queue #1: 80% +- Queue #3: 60% +- Queue #4: 70% +- Queue #5: 10% + +The Rx queues will be assigned to the cores in the following order:: + + Core 3: Q1 (80%) | + Core 7: Q4 (70%) | Q5 (10%) + Core 8: Q3 (60%) | Q0 (30%) + +To see the current measured usage history of PMD core cycles for each Rx +queue:: + + $ ovs-appctl dpif-netdev/pmd-rxq-show + +.. note:: + + A history of one minute is recorded and shown for each Rx queue to allow for + traffic pattern spikes. Any changes in the Rx queue's PMD core cycles usage, + due to traffic pattern or reconfig changes, will take one minute to be fully + reflected in the stats. + +Rx queue to PMD assignment takes place whenever there are configuration changes +or can be triggered by using:: + + $ ovs-appctl dpif-netdev/pmd-rxq-rebalance + +.. versionchanged:: 2.8.0 + + Automatic assignment of Rx queues to PMDs and the two related commands, + ``pmd-rxq-show`` and ``pmd-rxq-rebalance``, were added in OVS 2.8.0. Prior + to this, behavior was round-robin and processing cycles were not taken into + consideration. Tracking for stats was not available. + +.. versionchanged:: 2.9.0 + + The output of ``pmd-rxq-show`` was modified to include utilization as a + percentage. diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst index ca8a3289f..6f794f296 100644 --- a/Documentation/topics/dpdk/vhost-user.rst +++ b/Documentation/topics/dpdk/vhost-user.rst @@ -130,11 +130,10 @@ an additional set of parameters:: -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 -In addition, QEMU must allocate the VM's memory on hugetlbfs. vhost-user -ports access a virtio-net device's virtual rings and packet buffers mapping the -VM's physical memory on hugetlbfs. To enable vhost-user ports to map the VM's -memory into their process address space, pass the following parameters to -QEMU:: +In addition, QEMU must allocate the VM's memory on hugetlbfs. vhost-user ports +access a virtio-net device's virtual rings and packet buffers mapping the VM's +physical memory on hugetlbfs. To enable vhost-user ports to map the VM's memory +into their process address space, pass the following parameters to QEMU:: -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc @@ -154,18 +153,18 @@ where: The number of vectors, which is ``$q`` * 2 + 2 The vhost-user interface will be automatically reconfigured with required -number of rx and tx queues after connection of virtio device. Manual +number of Rx and Tx queues after connection of virtio device. Manual configuration of ``n_rxq`` is not supported because OVS will work properly only if ``n_rxq`` will match number of queues configured in QEMU. -A least 2 PMDs should be configured for the vswitch when using multiqueue. +A least two PMDs should be configured for the vswitch when using multiqueue. Using a single PMD will cause traffic to be enqueued to the same vhost queue rather than being distributed among different vhost queues for a vhost-user interface. If traffic destined for a VM configured with multiqueue arrives to the vswitch -via a physical DPDK port, then the number of rxqs should also be set to at -least 2 for that physical DPDK port. This is required to increase the +via a physical DPDK port, then the number of Rx queues should also be set to at +least two for that physical DPDK port. This is required to increase the probability that a different PMD will handle the multiqueue transmission to the guest using a different vhost queue.
This continues the breakup of the huge DPDK "howto" into smaller components. There are a couple of related changes included, such as using "Rx queue" instead of "rxq" and noting how Tx queues cannot be configured. Signed-off-by: Stephen Finucane <stephen@that.guru> --- v3: - Add new files to automake.mk - Add additional 'versionchanged' information for 'pmd-rxq-show' command v2: - Add cross-references from 'pmd' doc to 'vhost-user' and 'phy' docs - Add 'versionchanged' warning about automatic assignment of Rx queues - Add a 'todo' to describe Tx queue behavior --- Documentation/automake.mk | 1 + Documentation/howto/dpdk.rst | 86 ----------------- Documentation/topics/dpdk/index.rst | 1 + Documentation/topics/dpdk/phy.rst | 12 +++ Documentation/topics/dpdk/pmd.rst | 161 +++++++++++++++++++++++++++++++ Documentation/topics/dpdk/vhost-user.rst | 17 ++-- 6 files changed, 183 insertions(+), 95 deletions(-) create mode 100644 Documentation/topics/dpdk/pmd.rst