| Message ID | 1463171141-5677-1-git-send-email-mauricio.vasquezbernal@studenti.polito.it |
|---|---|
| State | Superseded |
| Headers | show |
> -----Original Message----- > From: Mauricio Vasquez B > [mailto:mauricio.vasquezbernal@studenti.polito.it] > Sent: Friday, May 13, 2016 9:26 PM > To: dev@openvswitch.org > Cc: aconole@redhat.com; lw@cn.fujitsu.com; mchandras@suse.de; Stokes, > Ian; fbl@sysclose.org; Traynor, Kevin; diproiettod@vmware.com > Subject: [PATCH v5] netdev-dpdk: add hotplug support > > In order to use dpdk ports in ovs they have to be bound to a DPDK > compatible driver before ovs is started. > > This patch adds the possibility to hotplug (or hot-unplug) a device > after ovs has been started. The implementation adds two appctl commands: > netdev-dpdk/port-attach and netdev-dpdk/port-detach > > After the user attaches a new device, it has to be added to a bridge > using the add-port command, similarly, before detaching a device, it has > to be removed using the del-port command. > > Signed-off-by: Mauricio Vasquez B > <mauricio.vasquezbernal@studenti.polito.it> > --- > v5: > - use two appctl commands instead of a single one > - rebase to master > v4: > - fix typo in commit message > - remove unnecessary whitespace change in INSTALL.DPDK.md > v3: > - create dpdk_port_attach and dpdk_port_detach functions > - modify mutex locking order > v2: > - use rte_eth_dev_is_valid_port() to check if a port is valid > INSTALL.DPDK.md | 24 +++++++++++++ > NEWS | 1 + > lib/netdev-dpdk.c | 101 > ++++++++++++++++++++++++++++++++++++++++++++++++++---- > 3 files changed, 119 insertions(+), 7 deletions(-) > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 93f92e4..8d5a0e0 > 100644 > --- a/INSTALL.DPDK.md > +++ b/INSTALL.DPDK.md > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: > For more details regarding egress-policer parameters please refer to > the > vswitch.xml. > > +9. Port Hotplug > + > + Ovs supports port hotplugging, it allows to use ports that were not > bound > + to DPDK when vswitchd was started. > + In order to attach a port, it has to be bound to DPDK using the > + dpdk_nic_bind.py script: > + > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` > + > + Then it can be attached to OVS: > + > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` > + > + At this point, the user can create a ovs port using the add-port > command. > + > + It is also possible to detach a port from ovs, the user has to > remove the > + port using the del-port command, then it can be detached using: > + > + `ovs-appctl netdev-dpdk/port-detach dpdk0` > + > + This feature is not supported by all the NICs, please refer to the > + [DPDK Port Hotplug Framework] in order to get more information. > + > Performance Tuning: > ------------------- > > @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. > [INSTALL.md]:INSTALL.md > [DPDK Linux GSG]: > http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and- > unbinding-network-ports-to-from-the-igb-uioor-vfio-modules > [DPDK Docs]: http://dpdk.org/doc > +[DPDK Port Hotplug Framework]: > +http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html > diff --git a/NEWS b/NEWS > index 4e81cad..d89d9a7 100644 > --- a/NEWS > +++ b/NEWS > @@ -32,6 +32,7 @@ Post-v2.5.0 > * DB entries have been added for many of the DPDK EAL command line > arguments. Additional arguments can be passed via the dpdk-extra > entry. > + * Port Hotplug is now supported. > - ovs-benchmark: This utility has been removed due to lack of use > and > bitrot. > - ovs-appctl: > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index > af86d19..6159a60 100644 > --- a/lib/netdev-dpdk.c > +++ b/lib/netdev-dpdk.c > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) > OVS_REQUIRES(dpdk_mutex) > int diag; > int n_rxq, n_txq; > > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { > return ENODEV; > } > > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn > *conn, int argc, > unixctl_command_reply(conn, "OK"); > } > > +static void > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, > + const char *argv[], void *aux OVS_UNUSED) { > + int ret; > + char response[128]; > + uint8_t port_id; > + > + ovs_mutex_lock(&dpdk_mutex); > + > + ret = rte_eth_dev_attach(argv[1], &port_id); > + if (ret < 0) { > + snprintf(response, sizeof(response), > + "Error attaching device '%s'", argv[1]); > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply_error(conn, response); > + return; > + } > + > + snprintf(response, sizeof(response), > + "Device '%s' has been attached as 'dpdk%d'", argv[1], > + port_id); > + > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply(conn, response); } > + > +static void > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, > + const char *argv[], void *aux OVS_UNUSED) { > + int ret; > + char response[128]; > + unsigned int parsed_port; > + uint8_t port_id; > + char devname[RTE_ETH_NAME_MAX_LEN]; > + > + ovs_mutex_lock(&dpdk_mutex); > + > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); > + if (ret) { > + snprintf(response, sizeof(response), > + "'%s' is not a valid port", argv[1]); > + goto error; > + } > + > + port_id = parsed_port; > + > + struct netdev *netdev = netdev_from_name(argv[1]); > + if (netdev) { > + netdev_close(netdev); > + snprintf(response, sizeof(response), > + "Port '%s' is being used. Remove it before detaching", > + argv[1]); > + goto error; > + } > + > + rte_eth_dev_close(port_id); > + > + ret = rte_eth_dev_detach(port_id, devname); > + if (ret < 0) { > + snprintf(response, sizeof(response), > + "Port '%s' can not be detached", argv[1]); > + goto error; > + } > + > + snprintf(response, sizeof(response), > + "Port '%s' has been detached", argv[1]); > + > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply(conn, response); > + return; > + > +error: > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply_error(conn, response); } > + > /* > * Set virtqueue flags so that we do not receive interrupts. > */ > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) > "[netdev] up|down", 1, 2, > netdev_dpdk_set_admin_state, NULL); > > + unixctl_command_register("netdev-dpdk/port-attach", > + "pci address of device", 1, 1, > + netdev_dpdk_port_attach, NULL); > + > + unixctl_command_register("netdev-dpdk/port-detach", > + "port", 1, 1, > + netdev_dpdk_port_detach, NULL); > + > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); > } > > /* Client Rings */ > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned > int port_no, { > struct dpdk_ring *ivshmem; > char ring_name[RTE_RING_NAMESIZE]; > - int err; > + int err, port_id; > > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); > if (ivshmem == NULL) { > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned > int port_no, > return ENOMEM; > } > > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > - &ivshmem->cring_tx, 1, SOCKET0); > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > + &ivshmem->cring_tx, 1, SOCKET0); > > - if (err < 0) { > + if (port_id < 0) { > rte_free(ivshmem); > return ENODEV; > } > > ivshmem->user_port_id = port_no; > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; > + ivshmem->eth_port_id = port_id; > + *eth_port_id = port_id; > + > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); > > - *eth_port_id = ivshmem->eth_port_id; > return 0; > } > > -- > 1.9.1 Thanks for the latest patch revision Mauricio. I've tested it with an Intel 82599 NIC and it works without issue. However I did come across an issue related to the XL710 interface when it is attached, detached and then re-attached, essentially the rte_init function fails as the device appears as busy. I think this issue is specific to the Xl710 however as the same behavior could not be replicated with the Intel 82599 NIC. As you have it called out in your notes that hot plug support is interface dependent I guess this is ok. It's seems to be a DPDK issue for the XL710 and will need further investigation separate to this work. Tested-by: ian.stokes@intel.com Thanks Ian
On Fri, May 13, 2016 at 10:25:41PM +0200, Mauricio Vasquez B wrote: > In order to use dpdk ports in ovs they have to be bound to a DPDK > compatible driver before ovs is started. > > This patch adds the possibility to hotplug (or hot-unplug) a device > after ovs has been started. The implementation adds two appctl commands: > netdev-dpdk/port-attach and netdev-dpdk/port-detach > > After the user attaches a new device, it has to be added to a bridge > using the add-port command, similarly, before detaching a device, > it has to be removed using the del-port command. > > Signed-off-by: Mauricio Vasquez B <mauricio.vasquezbernal@studenti.polito.it> The patch looks good, but it didn't work for me with VFIO (it's not supported by the DPDK hotplug framework). Since the doc includes instructions to use VFIO, should we add a note for the lack of hotplug support? The attach/detach operations seem to work with uio_pci_generic, but it doesn't move packets. The interface statistics shows only 'errs' increasing a lot and few 'drops'. Kernel: 4.5.0+ OVS: master (b396293a) + patch DPDK: 16.04 Drivers: vfio-pci or uio_pci_generic Hardware: 82599ES Do you know if there are patches/plans to support that with VFIO or uio_pci_generic? fbl > --- > v5: > - use two appctl commands instead of a single one > - rebase to master > v4: > - fix typo in commit message > - remove unnecessary whitespace change in INSTALL.DPDK.md > v3: > - create dpdk_port_attach and dpdk_port_detach functions > - modify mutex locking order > v2: > - use rte_eth_dev_is_valid_port() to check if a port is valid > INSTALL.DPDK.md | 24 +++++++++++++ > NEWS | 1 + > lib/netdev-dpdk.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++---- > 3 files changed, 119 insertions(+), 7 deletions(-) > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md > index 93f92e4..8d5a0e0 100644 > --- a/INSTALL.DPDK.md > +++ b/INSTALL.DPDK.md > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: > For more details regarding egress-policer parameters please refer to the > vswitch.xml. > > +9. Port Hotplug > + > + Ovs supports port hotplugging, it allows to use ports that were not bound > + to DPDK when vswitchd was started. > + In order to attach a port, it has to be bound to DPDK using the > + dpdk_nic_bind.py script: > + > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` > + > + Then it can be attached to OVS: > + > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` > + > + At this point, the user can create a ovs port using the add-port command. > + > + It is also possible to detach a port from ovs, the user has to remove the > + port using the del-port command, then it can be detached using: > + > + `ovs-appctl netdev-dpdk/port-detach dpdk0` > + > + This feature is not supported by all the NICs, please refer to the > + [DPDK Port Hotplug Framework] in order to get more information. > + > Performance Tuning: > ------------------- > > @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. > [INSTALL.md]:INSTALL.md > [DPDK Linux GSG]: http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules > [DPDK Docs]: http://dpdk.org/doc > +[DPDK Port Hotplug Framework]: http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html > diff --git a/NEWS b/NEWS > index 4e81cad..d89d9a7 100644 > --- a/NEWS > +++ b/NEWS > @@ -32,6 +32,7 @@ Post-v2.5.0 > * DB entries have been added for many of the DPDK EAL command line > arguments. Additional arguments can be passed via the dpdk-extra > entry. > + * Port Hotplug is now supported. > - ovs-benchmark: This utility has been removed due to lack of use and > bitrot. > - ovs-appctl: > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c > index af86d19..6159a60 100644 > --- a/lib/netdev-dpdk.c > +++ b/lib/netdev-dpdk.c > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) OVS_REQUIRES(dpdk_mutex) > int diag; > int n_rxq, n_txq; > > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { > return ENODEV; > } > > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn, int argc, > unixctl_command_reply(conn, "OK"); > } > > +static void > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, > + const char *argv[], void *aux OVS_UNUSED) > +{ > + int ret; > + char response[128]; > + uint8_t port_id; > + > + ovs_mutex_lock(&dpdk_mutex); > + > + ret = rte_eth_dev_attach(argv[1], &port_id); > + if (ret < 0) { > + snprintf(response, sizeof(response), > + "Error attaching device '%s'", argv[1]); > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply_error(conn, response); > + return; > + } > + > + snprintf(response, sizeof(response), > + "Device '%s' has been attached as 'dpdk%d'", argv[1], port_id); > + > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply(conn, response); > +} > + > +static void > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, > + const char *argv[], void *aux OVS_UNUSED) > +{ > + int ret; > + char response[128]; > + unsigned int parsed_port; > + uint8_t port_id; > + char devname[RTE_ETH_NAME_MAX_LEN]; > + > + ovs_mutex_lock(&dpdk_mutex); > + > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); > + if (ret) { > + snprintf(response, sizeof(response), > + "'%s' is not a valid port", argv[1]); > + goto error; > + } > + > + port_id = parsed_port; > + > + struct netdev *netdev = netdev_from_name(argv[1]); > + if (netdev) { > + netdev_close(netdev); > + snprintf(response, sizeof(response), > + "Port '%s' is being used. Remove it before detaching", > + argv[1]); > + goto error; > + } > + > + rte_eth_dev_close(port_id); > + > + ret = rte_eth_dev_detach(port_id, devname); > + if (ret < 0) { > + snprintf(response, sizeof(response), > + "Port '%s' can not be detached", argv[1]); > + goto error; > + } > + > + snprintf(response, sizeof(response), > + "Port '%s' has been detached", argv[1]); > + > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply(conn, response); > + return; > + > +error: > + ovs_mutex_unlock(&dpdk_mutex); > + unixctl_command_reply_error(conn, response); > +} > + > /* > * Set virtqueue flags so that we do not receive interrupts. > */ > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) > "[netdev] up|down", 1, 2, > netdev_dpdk_set_admin_state, NULL); > > + unixctl_command_register("netdev-dpdk/port-attach", > + "pci address of device", 1, 1, > + netdev_dpdk_port_attach, NULL); > + > + unixctl_command_register("netdev-dpdk/port-detach", > + "port", 1, 1, > + netdev_dpdk_port_detach, NULL); > + > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); > } > > /* Client Rings */ > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, > { > struct dpdk_ring *ivshmem; > char ring_name[RTE_RING_NAMESIZE]; > - int err; > + int err, port_id; > > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); > if (ivshmem == NULL) { > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, > return ENOMEM; > } > > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > - &ivshmem->cring_tx, 1, SOCKET0); > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > + &ivshmem->cring_tx, 1, SOCKET0); > > - if (err < 0) { > + if (port_id < 0) { > rte_free(ivshmem); > return ENODEV; > } > > ivshmem->user_port_id = port_no; > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; > + ivshmem->eth_port_id = port_id; > + *eth_port_id = port_id; > + > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); > > - *eth_port_id = ivshmem->eth_port_id; > return 0; > } > > -- > 1.9.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
On Mon, May 16, 2016 at 7:13 PM, Stokes, Ian <ian.stokes@intel.com> wrote: > > -----Original Message----- > > From: Mauricio Vasquez B > > [mailto:mauricio.vasquezbernal@studenti.polito.it] > > Sent: Friday, May 13, 2016 9:26 PM > > To: dev@openvswitch.org > > Cc: aconole@redhat.com; lw@cn.fujitsu.com; mchandras@suse.de; Stokes, > > Ian; fbl@sysclose.org; Traynor, Kevin; diproiettod@vmware.com > > Subject: [PATCH v5] netdev-dpdk: add hotplug support > > > > In order to use dpdk ports in ovs they have to be bound to a DPDK > > compatible driver before ovs is started. > > > > This patch adds the possibility to hotplug (or hot-unplug) a device > > after ovs has been started. The implementation adds two appctl commands: > > netdev-dpdk/port-attach and netdev-dpdk/port-detach > > > > After the user attaches a new device, it has to be added to a bridge > > using the add-port command, similarly, before detaching a device, it has > > to be removed using the del-port command. > > > > Signed-off-by: Mauricio Vasquez B > > <mauricio.vasquezbernal@studenti.polito.it> > > --- > > v5: > > - use two appctl commands instead of a single one > > - rebase to master > > v4: > > - fix typo in commit message > > - remove unnecessary whitespace change in INSTALL.DPDK.md > > v3: > > - create dpdk_port_attach and dpdk_port_detach functions > > - modify mutex locking order > > v2: > > - use rte_eth_dev_is_valid_port() to check if a port is valid > > INSTALL.DPDK.md | 24 +++++++++++++ > > NEWS | 1 + > > lib/netdev-dpdk.c | 101 > > ++++++++++++++++++++++++++++++++++++++++++++++++++---- > > 3 files changed, 119 insertions(+), 7 deletions(-) > > > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 93f92e4..8d5a0e0 > > 100644 > > --- a/INSTALL.DPDK.md > > +++ b/INSTALL.DPDK.md > > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: > > For more details regarding egress-policer parameters please refer to > > the > > vswitch.xml. > > > > +9. Port Hotplug > > + > > + Ovs supports port hotplugging, it allows to use ports that were not > > bound > > + to DPDK when vswitchd was started. > > + In order to attach a port, it has to be bound to DPDK using the > > + dpdk_nic_bind.py script: > > + > > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` > > + > > + Then it can be attached to OVS: > > + > > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` > > + > > + At this point, the user can create a ovs port using the add-port > > command. > > + > > + It is also possible to detach a port from ovs, the user has to > > remove the > > + port using the del-port command, then it can be detached using: > > + > > + `ovs-appctl netdev-dpdk/port-detach dpdk0` > > + > > + This feature is not supported by all the NICs, please refer to the > > + [DPDK Port Hotplug Framework] in order to get more information. > > + > > Performance Tuning: > > ------------------- > > > > @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. > > [INSTALL.md]:INSTALL.md > > [DPDK Linux GSG]: > > http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and- > > unbinding-network-ports-to-from-the-igb-uioor-vfio-modules > > [DPDK Docs]: http://dpdk.org/doc > > +[DPDK Port Hotplug Framework]: > > +http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html > > diff --git a/NEWS b/NEWS > > index 4e81cad..d89d9a7 100644 > > --- a/NEWS > > +++ b/NEWS > > @@ -32,6 +32,7 @@ Post-v2.5.0 > > * DB entries have been added for many of the DPDK EAL command line > > arguments. Additional arguments can be passed via the dpdk-extra > > entry. > > + * Port Hotplug is now supported. > > - ovs-benchmark: This utility has been removed due to lack of use > > and > > bitrot. > > - ovs-appctl: > > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index > > af86d19..6159a60 100644 > > --- a/lib/netdev-dpdk.c > > +++ b/lib/netdev-dpdk.c > > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) > > OVS_REQUIRES(dpdk_mutex) > > int diag; > > int n_rxq, n_txq; > > > > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { > > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { > > return ENODEV; > > } > > > > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn > > *conn, int argc, > > unixctl_command_reply(conn, "OK"); > > } > > > > +static void > > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) { > > + int ret; > > + char response[128]; > > + uint8_t port_id; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = rte_eth_dev_attach(argv[1], &port_id); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Error attaching device '%s'", argv[1]); > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); > > + return; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Device '%s' has been attached as 'dpdk%d'", argv[1], > > + port_id); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); } > > + > > +static void > > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) { > > + int ret; > > + char response[128]; > > + unsigned int parsed_port; > > + uint8_t port_id; > > + char devname[RTE_ETH_NAME_MAX_LEN]; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); > > + if (ret) { > > + snprintf(response, sizeof(response), > > + "'%s' is not a valid port", argv[1]); > > + goto error; > > + } > > + > > + port_id = parsed_port; > > + > > + struct netdev *netdev = netdev_from_name(argv[1]); > > + if (netdev) { > > + netdev_close(netdev); > > + snprintf(response, sizeof(response), > > + "Port '%s' is being used. Remove it before detaching", > > + argv[1]); > > + goto error; > > + } > > + > > + rte_eth_dev_close(port_id); > > + > > + ret = rte_eth_dev_detach(port_id, devname); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Port '%s' can not be detached", argv[1]); > > + goto error; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Port '%s' has been detached", argv[1]); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); > > + return; > > + > > +error: > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); } > > + > > /* > > * Set virtqueue flags so that we do not receive interrupts. > > */ > > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) > > "[netdev] up|down", 1, 2, > > netdev_dpdk_set_admin_state, NULL); > > > > + unixctl_command_register("netdev-dpdk/port-attach", > > + "pci address of device", 1, 1, > > + netdev_dpdk_port_attach, NULL); > > + > > + unixctl_command_register("netdev-dpdk/port-detach", > > + "port", 1, 1, > > + netdev_dpdk_port_detach, NULL); > > + > > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); > > } > > > > /* Client Rings */ > > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned > > int port_no, { > > struct dpdk_ring *ivshmem; > > char ring_name[RTE_RING_NAMESIZE]; > > - int err; > > + int err, port_id; > > > > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); > > if (ivshmem == NULL) { > > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned > > int port_no, > > return ENOMEM; > > } > > > > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > - &ivshmem->cring_tx, 1, SOCKET0); > > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > + &ivshmem->cring_tx, 1, SOCKET0); > > > > - if (err < 0) { > > + if (port_id < 0) { > > rte_free(ivshmem); > > return ENODEV; > > } > > > > ivshmem->user_port_id = port_no; > > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; > > + ivshmem->eth_port_id = port_id; > > + *eth_port_id = port_id; > > + > > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); > > > > - *eth_port_id = ivshmem->eth_port_id; > > return 0; > > } > > > > -- > > 1.9.1 > > Thanks for the latest patch revision Mauricio. > I've tested it with an Intel 82599 NIC and it works without issue. > > However I did come across an issue related to the XL710 interface when it > is attached, detached and then re-attached, essentially the rte_init > function fails as the device appears as busy. I think this issue is > specific to the Xl710 however as the same behavior could not be replicated > with the Intel 82599 NIC. > > As you have it called out in your notes that hot plug support is interface > dependent I guess this is ok. It's seems to be a DPDK issue for the XL710 > and will need further investigation separate to this work. > > Unfortunately I don't have any XL170 to test it here, if you want, you can use the testpmd DPDK application to test the hot-plug support and discuss results with the DPDK developers. > Tested-by: ian.stokes@intel.com > > Thanks > Ian > Thanks for testing!
On Tue, May 17, 2016 at 12:44 AM, Flavio Leitner <fbl@sysclose.org> wrote: > On Fri, May 13, 2016 at 10:25:41PM +0200, Mauricio Vasquez B wrote: > > In order to use dpdk ports in ovs they have to be bound to a DPDK > > compatible driver before ovs is started. > > > > This patch adds the possibility to hotplug (or hot-unplug) a device > > after ovs has been started. The implementation adds two appctl commands: > > netdev-dpdk/port-attach and netdev-dpdk/port-detach > > > > After the user attaches a new device, it has to be added to a bridge > > using the add-port command, similarly, before detaching a device, > > it has to be removed using the del-port command. > > > > Signed-off-by: Mauricio Vasquez B < > mauricio.vasquezbernal@studenti.polito.it> > > The patch looks good, but it didn't work for me with VFIO (it's > not supported by the DPDK hotplug framework). Since the doc > includes instructions to use VFIO, should we add a note for the > lack of hotplug support? > > Yes, we should. I'll wait few days to see if there is more feedback and I'll send a new version including that note. > The attach/detach operations seem to work with uio_pci_generic, > but it doesn't move packets. The interface statistics shows > only 'errs' increasing a lot and few 'drops'. > > Kernel: 4.5.0+ > OVS: master (b396293a) + patch > DPDK: 16.04 > Drivers: vfio-pci or uio_pci_generic > Hardware: 82599ES > > I tried uio_pci_generic in the following conditions without any problem. Kernel: 3.19.0-32-generic OVS: master(d640a7e1) + patch DPDK: 16.04 Hardware: X540-AT2 > Do you know if there are patches/plans to support that with VFIO or > uio_pci_generic? > I am not aware of any plans for it. Thanks for reviewing Flavio! > fbl > > > > --- > > v5: > > - use two appctl commands instead of a single one > > - rebase to master > > v4: > > - fix typo in commit message > > - remove unnecessary whitespace change in INSTALL.DPDK.md > > v3: > > - create dpdk_port_attach and dpdk_port_detach functions > > - modify mutex locking order > > v2: > > - use rte_eth_dev_is_valid_port() to check if a port is valid > > INSTALL.DPDK.md | 24 +++++++++++++ > > NEWS | 1 + > > lib/netdev-dpdk.c | 101 > ++++++++++++++++++++++++++++++++++++++++++++++++++---- > > 3 files changed, 119 insertions(+), 7 deletions(-) > > > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md > > index 93f92e4..8d5a0e0 100644 > > --- a/INSTALL.DPDK.md > > +++ b/INSTALL.DPDK.md > > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: > > For more details regarding egress-policer parameters please refer to > the > > vswitch.xml. > > > > +9. Port Hotplug > > + > > + Ovs supports port hotplugging, it allows to use ports that were not > bound > > + to DPDK when vswitchd was started. > > + In order to attach a port, it has to be bound to DPDK using the > > + dpdk_nic_bind.py script: > > + > > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` > > + > > + Then it can be attached to OVS: > > + > > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` > > + > > + At this point, the user can create a ovs port using the add-port > command. > > + > > + It is also possible to detach a port from ovs, the user has to > remove the > > + port using the del-port command, then it can be detached using: > > + > > + `ovs-appctl netdev-dpdk/port-detach dpdk0` > > + > > + This feature is not supported by all the NICs, please refer to the > > + [DPDK Port Hotplug Framework] in order to get more information. > > + > > Performance Tuning: > > ------------------- > > > > @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. > > [INSTALL.md]:INSTALL.md > > [DPDK Linux GSG]: > http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules > > [DPDK Docs]: http://dpdk.org/doc > > +[DPDK Port Hotplug Framework]: > http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html > > diff --git a/NEWS b/NEWS > > index 4e81cad..d89d9a7 100644 > > --- a/NEWS > > +++ b/NEWS > > @@ -32,6 +32,7 @@ Post-v2.5.0 > > * DB entries have been added for many of the DPDK EAL command line > > arguments. Additional arguments can be passed via the dpdk-extra > > entry. > > + * Port Hotplug is now supported. > > - ovs-benchmark: This utility has been removed due to lack of use and > > bitrot. > > - ovs-appctl: > > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c > > index af86d19..6159a60 100644 > > --- a/lib/netdev-dpdk.c > > +++ b/lib/netdev-dpdk.c > > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) > OVS_REQUIRES(dpdk_mutex) > > int diag; > > int n_rxq, n_txq; > > > > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { > > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { > > return ENODEV; > > } > > > > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn > *conn, int argc, > > unixctl_command_reply(conn, "OK"); > > } > > > > +static void > > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) > > +{ > > + int ret; > > + char response[128]; > > + uint8_t port_id; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = rte_eth_dev_attach(argv[1], &port_id); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Error attaching device '%s'", argv[1]); > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); > > + return; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Device '%s' has been attached as 'dpdk%d'", argv[1], > port_id); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); > > +} > > + > > +static void > > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) > > +{ > > + int ret; > > + char response[128]; > > + unsigned int parsed_port; > > + uint8_t port_id; > > + char devname[RTE_ETH_NAME_MAX_LEN]; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); > > + if (ret) { > > + snprintf(response, sizeof(response), > > + "'%s' is not a valid port", argv[1]); > > + goto error; > > + } > > + > > + port_id = parsed_port; > > + > > + struct netdev *netdev = netdev_from_name(argv[1]); > > + if (netdev) { > > + netdev_close(netdev); > > + snprintf(response, sizeof(response), > > + "Port '%s' is being used. Remove it before detaching", > > + argv[1]); > > + goto error; > > + } > > + > > + rte_eth_dev_close(port_id); > > + > > + ret = rte_eth_dev_detach(port_id, devname); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Port '%s' can not be detached", argv[1]); > > + goto error; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Port '%s' has been detached", argv[1]); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); > > + return; > > + > > +error: > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); > > +} > > + > > /* > > * Set virtqueue flags so that we do not receive interrupts. > > */ > > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) > > "[netdev] up|down", 1, 2, > > netdev_dpdk_set_admin_state, NULL); > > > > + unixctl_command_register("netdev-dpdk/port-attach", > > + "pci address of device", 1, 1, > > + netdev_dpdk_port_attach, NULL); > > + > > + unixctl_command_register("netdev-dpdk/port-detach", > > + "port", 1, 1, > > + netdev_dpdk_port_detach, NULL); > > + > > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); > > } > > > > /* Client Rings */ > > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned > int port_no, > > { > > struct dpdk_ring *ivshmem; > > char ring_name[RTE_RING_NAMESIZE]; > > - int err; > > + int err, port_id; > > > > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); > > if (ivshmem == NULL) { > > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned > int port_no, > > return ENOMEM; > > } > > > > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > - &ivshmem->cring_tx, 1, SOCKET0); > > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > + &ivshmem->cring_tx, 1, SOCKET0); > > > > - if (err < 0) { > > + if (port_id < 0) { > > rte_free(ivshmem); > > return ENODEV; > > } > > > > ivshmem->user_port_id = port_no; > > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; > > + ivshmem->eth_port_id = port_id; > > + *eth_port_id = port_id; > > + > > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); > > > > - *eth_port_id = ivshmem->eth_port_id; > > return 0; > > } > > > > -- > > 1.9.1 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > http://openvswitch.org/mailman/listinfo/dev > > -- > fbl > >
Hi, I just sent v6: http://openvswitch.org/pipermail/dev/2016-May/071786.html On Thu, May 19, 2016 at 9:30 AM, Mauricio Vásquez < mauricio.vasquezbernal@studenti.polito.it> wrote: > > > On Tue, May 17, 2016 at 12:44 AM, Flavio Leitner <fbl@sysclose.org> wrote: > >> On Fri, May 13, 2016 at 10:25:41PM +0200, Mauricio Vasquez B wrote: >> > In order to use dpdk ports in ovs they have to be bound to a DPDK >> > compatible driver before ovs is started. >> > >> > This patch adds the possibility to hotplug (or hot-unplug) a device >> > after ovs has been started. The implementation adds two appctl commands: >> > netdev-dpdk/port-attach and netdev-dpdk/port-detach >> > >> > After the user attaches a new device, it has to be added to a bridge >> > using the add-port command, similarly, before detaching a device, >> > it has to be removed using the del-port command. >> > >> > Signed-off-by: Mauricio Vasquez B < >> mauricio.vasquezbernal@studenti.polito.it> >> >> The patch looks good, but it didn't work for me with VFIO (it's >> not supported by the DPDK hotplug framework). Since the doc >> includes instructions to use VFIO, should we add a note for the >> lack of hotplug support? >> >> > Yes, we should. I'll wait few days to see if there is more feedback and > I'll send a new version including that note. > > >> The attach/detach operations seem to work with uio_pci_generic, >> but it doesn't move packets. The interface statistics shows >> only 'errs' increasing a lot and few 'drops'. >> >> Kernel: 4.5.0+ >> OVS: master (b396293a) + patch >> DPDK: 16.04 >> Drivers: vfio-pci or uio_pci_generic >> Hardware: 82599ES >> >> > I tried uio_pci_generic in the following conditions without any problem. > > Kernel: 3.19.0-32-generic > OVS: master(d640a7e1) + patch > DPDK: 16.04 > Hardware: X540-AT2 > > >> Do you know if there are patches/plans to support that with VFIO or >> uio_pci_generic? >> > > I am not aware of any plans for it. > > Thanks for reviewing Flavio! > > >> fbl >> >> >> > --- >> > v5: >> > - use two appctl commands instead of a single one >> > - rebase to master >> > v4: >> > - fix typo in commit message >> > - remove unnecessary whitespace change in INSTALL.DPDK.md >> > v3: >> > - create dpdk_port_attach and dpdk_port_detach functions >> > - modify mutex locking order >> > v2: >> > - use rte_eth_dev_is_valid_port() to check if a port is valid >> > INSTALL.DPDK.md | 24 +++++++++++++ >> > NEWS | 1 + >> > lib/netdev-dpdk.c | 101 >> ++++++++++++++++++++++++++++++++++++++++++++++++++---- >> > 3 files changed, 119 insertions(+), 7 deletions(-) >> > >> > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md >> > index 93f92e4..8d5a0e0 100644 >> > --- a/INSTALL.DPDK.md >> > +++ b/INSTALL.DPDK.md >> > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: >> > For more details regarding egress-policer parameters please refer >> to the >> > vswitch.xml. >> > >> > +9. Port Hotplug >> > + >> > + Ovs supports port hotplugging, it allows to use ports that were not >> bound >> > + to DPDK when vswitchd was started. >> > + In order to attach a port, it has to be bound to DPDK using the >> > + dpdk_nic_bind.py script: >> > + >> > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` >> > + >> > + Then it can be attached to OVS: >> > + >> > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` >> > + >> > + At this point, the user can create a ovs port using the add-port >> command. >> > + >> > + It is also possible to detach a port from ovs, the user has to >> remove the >> > + port using the del-port command, then it can be detached using: >> > + >> > + `ovs-appctl netdev-dpdk/port-detach dpdk0` >> > + >> > + This feature is not supported by all the NICs, please refer to the >> > + [DPDK Port Hotplug Framework] in order to get more information. >> > + >> > Performance Tuning: >> > ------------------- >> > >> > @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. >> > [INSTALL.md]:INSTALL.md >> > [DPDK Linux GSG]: >> http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules >> > [DPDK Docs]: http://dpdk.org/doc >> > +[DPDK Port Hotplug Framework]: >> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html >> > diff --git a/NEWS b/NEWS >> > index 4e81cad..d89d9a7 100644 >> > --- a/NEWS >> > +++ b/NEWS >> > @@ -32,6 +32,7 @@ Post-v2.5.0 >> > * DB entries have been added for many of the DPDK EAL command line >> > arguments. Additional arguments can be passed via the dpdk-extra >> > entry. >> > + * Port Hotplug is now supported. >> > - ovs-benchmark: This utility has been removed due to lack of use >> and >> > bitrot. >> > - ovs-appctl: >> > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c >> > index af86d19..6159a60 100644 >> > --- a/lib/netdev-dpdk.c >> > +++ b/lib/netdev-dpdk.c >> > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) >> OVS_REQUIRES(dpdk_mutex) >> > int diag; >> > int n_rxq, n_txq; >> > >> > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { >> > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { >> > return ENODEV; >> > } >> > >> > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn >> *conn, int argc, >> > unixctl_command_reply(conn, "OK"); >> > } >> > >> > +static void >> > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, >> > + const char *argv[], void *aux OVS_UNUSED) >> > +{ >> > + int ret; >> > + char response[128]; >> > + uint8_t port_id; >> > + >> > + ovs_mutex_lock(&dpdk_mutex); >> > + >> > + ret = rte_eth_dev_attach(argv[1], &port_id); >> > + if (ret < 0) { >> > + snprintf(response, sizeof(response), >> > + "Error attaching device '%s'", argv[1]); >> > + ovs_mutex_unlock(&dpdk_mutex); >> > + unixctl_command_reply_error(conn, response); >> > + return; >> > + } >> > + >> > + snprintf(response, sizeof(response), >> > + "Device '%s' has been attached as 'dpdk%d'", argv[1], >> port_id); >> > + >> > + ovs_mutex_unlock(&dpdk_mutex); >> > + unixctl_command_reply(conn, response); >> > +} >> > + >> > +static void >> > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, >> > + const char *argv[], void *aux OVS_UNUSED) >> > +{ >> > + int ret; >> > + char response[128]; >> > + unsigned int parsed_port; >> > + uint8_t port_id; >> > + char devname[RTE_ETH_NAME_MAX_LEN]; >> > + >> > + ovs_mutex_lock(&dpdk_mutex); >> > + >> > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); >> > + if (ret) { >> > + snprintf(response, sizeof(response), >> > + "'%s' is not a valid port", argv[1]); >> > + goto error; >> > + } >> > + >> > + port_id = parsed_port; >> > + >> > + struct netdev *netdev = netdev_from_name(argv[1]); >> > + if (netdev) { >> > + netdev_close(netdev); >> > + snprintf(response, sizeof(response), >> > + "Port '%s' is being used. Remove it before detaching", >> > + argv[1]); >> > + goto error; >> > + } >> > + >> > + rte_eth_dev_close(port_id); >> > + >> > + ret = rte_eth_dev_detach(port_id, devname); >> > + if (ret < 0) { >> > + snprintf(response, sizeof(response), >> > + "Port '%s' can not be detached", argv[1]); >> > + goto error; >> > + } >> > + >> > + snprintf(response, sizeof(response), >> > + "Port '%s' has been detached", argv[1]); >> > + >> > + ovs_mutex_unlock(&dpdk_mutex); >> > + unixctl_command_reply(conn, response); >> > + return; >> > + >> > +error: >> > + ovs_mutex_unlock(&dpdk_mutex); >> > + unixctl_command_reply_error(conn, response); >> > +} >> > + >> > /* >> > * Set virtqueue flags so that we do not receive interrupts. >> > */ >> > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) >> > "[netdev] up|down", 1, 2, >> > netdev_dpdk_set_admin_state, NULL); >> > >> > + unixctl_command_register("netdev-dpdk/port-attach", >> > + "pci address of device", 1, 1, >> > + netdev_dpdk_port_attach, NULL); >> > + >> > + unixctl_command_register("netdev-dpdk/port-detach", >> > + "port", 1, 1, >> > + netdev_dpdk_port_detach, NULL); >> > + >> > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); >> > } >> > >> > /* Client Rings */ >> > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned >> int port_no, >> > { >> > struct dpdk_ring *ivshmem; >> > char ring_name[RTE_RING_NAMESIZE]; >> > - int err; >> > + int err, port_id; >> > >> > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); >> > if (ivshmem == NULL) { >> > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], >> unsigned int port_no, >> > return ENOMEM; >> > } >> > >> > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, >> > - &ivshmem->cring_tx, 1, SOCKET0); >> > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, >> > + &ivshmem->cring_tx, 1, SOCKET0); >> > >> > - if (err < 0) { >> > + if (port_id < 0) { >> > rte_free(ivshmem); >> > return ENODEV; >> > } >> > >> > ivshmem->user_port_id = port_no; >> > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; >> > + ivshmem->eth_port_id = port_id; >> > + *eth_port_id = port_id; >> > + >> > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); >> > >> > - *eth_port_id = ivshmem->eth_port_id; >> > return 0; >> > } >> > >> > -- >> > 1.9.1 >> > >> > _______________________________________________ >> > dev mailing list >> > dev@openvswitch.org >> > http://openvswitch.org/mailman/listinfo/dev >> >> -- >> fbl >> >> >
diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 93f92e4..8d5a0e0 100644 --- a/INSTALL.DPDK.md +++ b/INSTALL.DPDK.md @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: For more details regarding egress-policer parameters please refer to the vswitch.xml. +9. Port Hotplug + + Ovs supports port hotplugging, it allows to use ports that were not bound + to DPDK when vswitchd was started. + In order to attach a port, it has to be bound to DPDK using the + dpdk_nic_bind.py script: + + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` + + Then it can be attached to OVS: + + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` + + At this point, the user can create a ovs port using the add-port command. + + It is also possible to detach a port from ovs, the user has to remove the + port using the del-port command, then it can be detached using: + + `ovs-appctl netdev-dpdk/port-detach dpdk0` + + This feature is not supported by all the NICs, please refer to the + [DPDK Port Hotplug Framework] in order to get more information. + Performance Tuning: ------------------- @@ -999,3 +1022,4 @@ Please report problems to bugs@openvswitch.org. [INSTALL.md]:INSTALL.md [DPDK Linux GSG]: http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules [DPDK Docs]: http://dpdk.org/doc +[DPDK Port Hotplug Framework]: http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html diff --git a/NEWS b/NEWS index 4e81cad..d89d9a7 100644 --- a/NEWS +++ b/NEWS @@ -32,6 +32,7 @@ Post-v2.5.0 * DB entries have been added for many of the DPDK EAL command line arguments. Additional arguments can be passed via the dpdk-extra entry. + * Port Hotplug is now supported. - ovs-benchmark: This utility has been removed due to lack of use and bitrot. - ovs-appctl: diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index af86d19..6159a60 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) OVS_REQUIRES(dpdk_mutex) int diag; int n_rxq, n_txq; - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { + if (!rte_eth_dev_is_valid_port(dev->port_id)) { return ENODEV; } @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn, int argc, unixctl_command_reply(conn, "OK"); } +static void +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *aux OVS_UNUSED) +{ + int ret; + char response[128]; + uint8_t port_id; + + ovs_mutex_lock(&dpdk_mutex); + + ret = rte_eth_dev_attach(argv[1], &port_id); + if (ret < 0) { + snprintf(response, sizeof(response), + "Error attaching device '%s'", argv[1]); + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply_error(conn, response); + return; + } + + snprintf(response, sizeof(response), + "Device '%s' has been attached as 'dpdk%d'", argv[1], port_id); + + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply(conn, response); +} + +static void +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *aux OVS_UNUSED) +{ + int ret; + char response[128]; + unsigned int parsed_port; + uint8_t port_id; + char devname[RTE_ETH_NAME_MAX_LEN]; + + ovs_mutex_lock(&dpdk_mutex); + + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); + if (ret) { + snprintf(response, sizeof(response), + "'%s' is not a valid port", argv[1]); + goto error; + } + + port_id = parsed_port; + + struct netdev *netdev = netdev_from_name(argv[1]); + if (netdev) { + netdev_close(netdev); + snprintf(response, sizeof(response), + "Port '%s' is being used. Remove it before detaching", + argv[1]); + goto error; + } + + rte_eth_dev_close(port_id); + + ret = rte_eth_dev_detach(port_id, devname); + if (ret < 0) { + snprintf(response, sizeof(response), + "Port '%s' can not be detached", argv[1]); + goto error; + } + + snprintf(response, sizeof(response), + "Port '%s' has been detached", argv[1]); + + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply(conn, response); + return; + +error: + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply_error(conn, response); +} + /* * Set virtqueue flags so that we do not receive interrupts. */ @@ -2404,6 +2481,15 @@ dpdk_common_init(void) "[netdev] up|down", 1, 2, netdev_dpdk_set_admin_state, NULL); + unixctl_command_register("netdev-dpdk/port-attach", + "pci address of device", 1, 1, + netdev_dpdk_port_attach, NULL); + + unixctl_command_register("netdev-dpdk/port-detach", + "port", 1, 1, + netdev_dpdk_port_detach, NULL); + + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); } /* Client Rings */ @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, { struct dpdk_ring *ivshmem; char ring_name[RTE_RING_NAMESIZE]; - int err; + int err, port_id; ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); if (ivshmem == NULL) { @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, return ENOMEM; } - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, - &ivshmem->cring_tx, 1, SOCKET0); + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, + &ivshmem->cring_tx, 1, SOCKET0); - if (err < 0) { + if (port_id < 0) { rte_free(ivshmem); return ENODEV; } ivshmem->user_port_id = port_no; - ivshmem->eth_port_id = rte_eth_dev_count() - 1; + ivshmem->eth_port_id = port_id; + *eth_port_id = port_id; + ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); - *eth_port_id = ivshmem->eth_port_id; return 0; }
In order to use dpdk ports in ovs they have to be bound to a DPDK compatible driver before ovs is started. This patch adds the possibility to hotplug (or hot-unplug) a device after ovs has been started. The implementation adds two appctl commands: netdev-dpdk/port-attach and netdev-dpdk/port-detach After the user attaches a new device, it has to be added to a bridge using the add-port command, similarly, before detaching a device, it has to be removed using the del-port command. Signed-off-by: Mauricio Vasquez B <mauricio.vasquezbernal@studenti.polito.it> --- v5: - use two appctl commands instead of a single one - rebase to master v4: - fix typo in commit message - remove unnecessary whitespace change in INSTALL.DPDK.md v3: - create dpdk_port_attach and dpdk_port_detach functions - modify mutex locking order v2: - use rte_eth_dev_is_valid_port() to check if a port is valid INSTALL.DPDK.md | 24 +++++++++++++ NEWS | 1 + lib/netdev-dpdk.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 119 insertions(+), 7 deletions(-)