From patchwork Wed Jun 12 14:32:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 1946959 X-Patchwork-Delegate: ktraynor@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=avDfXxmj; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Vzp1d5w1Dz1yXv for ; Thu, 13 Jun 2024 00:33:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E1F3D4116B; Wed, 12 Jun 2024 14:33:23 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 5AvH9shDD0rX; Wed, 12 Jun 2024 14:33:22 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 617964119B Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=avDfXxmj Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 617964119B; Wed, 12 Jun 2024 14:33:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A7ACCC0012; Wed, 12 Jun 2024 14:33:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3D0E3C0011 for ; Wed, 12 Jun 2024 14:33:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1E15B82A26 for ; Wed, 12 Jun 2024 14:33:20 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id bWvsN9eRiSBh for ; Wed, 12 Jun 2024 14:33:19 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=ktraynor@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 57BAF820EE Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 57BAF820EE Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=avDfXxmj Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 57BAF820EE for ; Wed, 12 Jun 2024 14:33:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718202797; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I1X3QZPYzGKxw7IgvU1n9lsNqTCETm4PSiAvi0KWBVE=; b=avDfXxmjrj6rWQM81vVTVtWZhDtrj8Z/4+7f1hQAm/P5ElxmZTbRKzwVptY+j2XwQZI/mc 98ny3V+2wEEQGvgWmX9rburzdMCdrEWerj01jS4ItWXma9H1IQ/bi/eUmWYSKRCzehApuA OP/wCoM54WJOzupGUPYumcxB51d4LtQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-177-Qr-HL_h0OiSduzQU8KlxsQ-1; Wed, 12 Jun 2024 10:33:16 -0400 X-MC-Unique: Qr-HL_h0OiSduzQU8KlxsQ-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DC70219560A1 for ; Wed, 12 Jun 2024 14:33:15 +0000 (UTC) Received: from rh.redhat.com (unknown [10.39.192.36]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 276901956050; Wed, 12 Jun 2024 14:33:13 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org Date: Wed, 12 Jun 2024 15:32:55 +0100 Message-ID: <20240612143255.63973-2-ktraynor@redhat.com> In-Reply-To: <20240612143255.63973-1-ktraynor@redhat.com> References: <20240612143255.63973-1-ktraynor@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH 1/1] netdev-dpdk: Check pending reset when adding device. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.marchand@redhat.com Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When a device reset interrupt event (RTE_ETH_EVENT_INTR_RESET) is detected for a DPDK device added to OVS, a device reset is performed. If a device reset interrupt event is detected for a device before it is added to OVS, device reset is not called. If that device is later attempted to be added to OVS, it may fail while being configured if it is still pending a reset as pending reset is not checked when adding a device. A simple way to force a reset event from the ice driver for an iavf device is to set the mac address after binding iavf dev to vfio but before adding to OVS. (note: should not be set like this in normal case). e.g. $ echo 2 > /sys/class/net/ens3f0/device/sriov_numvfs $ ./devbind.py -b vfio-pci 0000:d8:01.1 $ ip link set ens3f0 vf 1 mac 26:ab:e6:6f:79:4d $ ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk \ options:dpdk-devargs=0000:d8:01.1 |dpdk|ERR|Port1 dev_configure = -1 |netdev_dpdk|WARN|Interface dpdk0 eth_dev setup error Operation not permitted |netdev_dpdk|ERR|Interface dpdk0(rxq:1 txq:5 lsc interrupt mode:false) configure error: Operation not permitted |dpif_netdev|ERR|Failed to set interface dpdk0 new configuration Add a check if there was any previous device reset interrupt events when a device is added to OVS. If there was, perform the reset before continuing with the rest of the configuration. netdev_dpdk_pending_reset[] already tracks device reset interrupt events for all devices, so it can be reused to check if there is a reset needed during configuration of newly added devices. By extending it's usage, dev->reset_needed is no longer needed. Signed-off-by: Kevin Traynor Reviewed-by: David Marchand --- lib/netdev-dpdk.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 0fa37d514..9ceb0d972 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -465,7 +465,6 @@ struct netdev_dpdk { /* If true, rte_eth_dev_start() was successfully called */ bool started; - bool reset_needed; - /* 1 pad byte here. */ struct eth_addr hwaddr; + /* 2 pad byte here. */ int mtu; int socket_id; @@ -1532,5 +1531,4 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->attached = false; dev->started = false; - dev->reset_needed = false; ovsrcu_init(&dev->qos_conf, NULL); @@ -2155,5 +2153,4 @@ netdev_dpdk_run(const struct netdev_class *netdev_class OVS_UNUSED) continue; } - atomic_store_relaxed(&netdev_dpdk_pending_reset[port_id], false); ovs_mutex_lock(&dpdk_mutex); @@ -2161,5 +2158,4 @@ netdev_dpdk_run(const struct netdev_class *netdev_class OVS_UNUSED) if (dev) { ovs_mutex_lock(&dev->mutex); - dev->reset_needed = true; netdev_request_reconfigure(&dev->up); VLOG_DBG_RL(&rl, "%s: Device reset requested.", @@ -6073,4 +6069,5 @@ netdev_dpdk_reconfigure(struct netdev *netdev) { struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + bool pending_reset; bool try_rx_steer; int err = 0; @@ -6084,4 +6081,7 @@ netdev_dpdk_reconfigure(struct netdev *netdev) } + atomic_read_relaxed(&netdev_dpdk_pending_reset[dev->port_id], + &pending_reset); + if (netdev->n_txq == dev->requested_n_txq && netdev->n_rxq == dev->requested_n_rxq @@ -6093,5 +6093,5 @@ netdev_dpdk_reconfigure(struct netdev *netdev) && eth_addr_equals(dev->hwaddr, dev->requested_hwaddr) && dev->socket_id == dev->requested_socket_id - && dev->started && !dev->reset_needed) { + && dev->started && !pending_reset) { /* Reconfiguration is unnecessary */ @@ -6102,8 +6102,12 @@ retry: dpdk_rx_steer_unconfigure(dev); - if (dev->reset_needed) { + if (pending_reset) { + /* + * Set false before reset to avoid missing a new reset interrupt event + * in a race with event callback. + */ + atomic_store_relaxed(&netdev_dpdk_pending_reset[dev->port_id], false); rte_eth_dev_reset(dev->port_id); if_notifier_manual_report(); - dev->reset_needed = false; } else { rte_eth_dev_stop(dev->port_id);