From patchwork Mon Oct 28 10:03:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Swiatkowski X-Patchwork-Id: 2003190 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=osuosl.org header.i=@osuosl.org header.a=rsa-sha256 header.s=default header.b=B6N4160W; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XcTVy0CVTz1xw0 for ; Mon, 28 Oct 2024 21:03:53 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 357D5407A9; Mon, 28 Oct 2024 10:03:52 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id nxGOmnGWAg76; Mon, 28 Oct 2024 10:03:50 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.142; helo=lists1.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 882DE4079B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1730109830; bh=XHCOPi9q96E3YhaZ+DH9dpu5yNhWUuvDcP8qeZW5Mk0=; h=From:To:Cc:Date:Subject:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From; b=B6N4160WsyF8E+Q36Z0SCJi3E6q3F5GY3zmWGVIL506kkFJ78JqFa3ApNTI7a83Ou Eh1iV/VZcPcUveXLupHjQtiL57qE37+HstR91QkdW1Ia6IsqFh5oSSBeweoKo1Y4d3 v483FGS48NRyMiRwmuN8p9hBVDeCRzkHJQZvAr8hpxvLZPk6+XGlNjz6EN8F9uRkOX a+RgFvnTdwOyhyWqeMydMJadrkTIP+B7qo4zxjSEuFFtIf1ECQRzSWInxwGcax+la0 2sZF5U5Csz/BYjt5ImtpPqpb79SkK9+UDzRxnsZRxz+u2DkGg+jUYkkSvnYB1D+02o q0kf+hu+kg91A== Received: from lists1.osuosl.org (lists1.osuosl.org [140.211.166.142]) by smtp4.osuosl.org (Postfix) with ESMTP id 882DE4079B; Mon, 28 Oct 2024 10:03:50 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists1.osuosl.org (Postfix) with ESMTP id D02D5112A for ; Mon, 28 Oct 2024 10:03:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id BEA3781A3E for ; Mon, 28 Oct 2024 10:03:48 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ZXXY-ZdtSz4w for ; Mon, 28 Oct 2024 10:03:47 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=198.175.65.19; helo=mgamail.intel.com; envelope-from=michal.swiatkowski@linux.intel.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 6D5C481A39 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 6D5C481A39 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6D5C481A39 for ; Mon, 28 Oct 2024 10:03:47 +0000 (UTC) X-CSE-ConnectionGUID: 7t8hJZAgSCugCI9D0nNDtA== X-CSE-MsgGUID: XwpZ+CJbTvCkj6MlqGcr0w== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="29560707" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="29560707" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Oct 2024 03:03:46 -0700 X-CSE-ConnectionGUID: cOVrWze/R++25KZk1tn0Ig== X-CSE-MsgGUID: vfhcsBAxR2y/Cg0tPcyp+g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,238,1725346800"; d="scan'208";a="86182268" Received: from gk3153-dr2-r750-36946.igk.intel.com ([10.102.20.192]) by fmviesa004.fm.intel.com with ESMTP; 28 Oct 2024 03:03:42 -0700 From: Michal Swiatkowski To: intel-wired-lan@lists.osuosl.org Cc: netdev@vger.kernel.org, pawel.chmielewski@intel.com, sridhar.samudrala@intel.com, jacob.e.keller@intel.com, pio.raczynski@gmail.com, konrad.knitter@intel.com, marcin.szycik@intel.com, wojciech.drewek@intel.com, nex.sw.ncis.nat.hpm.dev@intel.com, przemyslaw.kitszel@intel.com, jiri@resnulli.us, horms@kernel.org, David.Laight@ACULAB.COM Date: Mon, 28 Oct 2024 11:03:32 +0100 Message-ID: <20241028100341.16631-1-michal.swiatkowski@linux.intel.com> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730109828; x=1761645828; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=af/JTI5FMLotcKPq0HC4amBInS1wpiR0Ry9ax+mSdkU=; b=d/QQVALVWf0vRQz0sExpBOt/8aP6Ey22YGCrEmIRTo4bb5/gCg1Xlvm7 02vVo5Ym/Pt6jq0BuK6qKucqCTMK89NW7y7zEYt63zk1C0f/OCX6MVkgL StD3vK7TeMDBH0CP2M7wWuWHhD9LYfxTLNzIQ+ekBfan2KcsZd9IABnbN F9T9hVSfXSIppOE/erWKxB9gDelutGMVpEdZvvbQpFtitlH4Eq4+8fE7T WTcsn9/T5cb6f6k8hE6HoTZt66EekwSIkWSs5C1k/goHAaPpvOnPo4scM YMgrZBVWeIrD9HRu8va2EBduf0TlLpgcWnYfp2f/DNtJ0ggOleFLB1C9s Q==; X-Mailman-Original-Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=linux.intel.com X-Mailman-Original-Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=d/QQVALV Subject: [Intel-wired-lan] [iwl-next v6 0/9] ice: managing MSI-X in driver X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" Hi, It is another try to allow user to manage amount of MSI-X used for each feature in ice. First was via devlink resources API, it wasn't accepted in upstream. Also static MSI-X allocation using devlink resources isn't really user friendly. This try is using more dynamic way. "Dynamic" across whole kernel when platform supports it and "dynamic" across the driver when not. To achieve that reuse global devlink parameter pf_msix_max and pf_msix_min. It fits how ice hardware counts MSI-X. In case of ice amount of MSI-X reported on PCI is a whole MSI-X for the card (with MSI-X for VFs also). Having pf_msix_max allow user to statically set how many MSI-X he wants on PF and how many should be reserved for VFs. pf_msix_min is used to set minimum number of MSI-X with which ice driver should probe correctly. Meaning of this field in case of dynamic vs static allocation: - on system with dynamic MSI-X allocation support * alloc pf_msix_min as static, rest will be allocated dynamically - on system without dynamic MSI-X allocation support * try alloc pf_msix_max as static, minimum acceptable result is pf_msix_min As Jesse and Piotr suggested pf_msix_max and pf_msix_min can (an probably should) be stored in NVM. This patchset isn't implementing that. Dynamic (kernel or driver) way means that splitting MSI-X across the RDMA and eth in case there is a MSI-X shortage isn't correct. Can work when dynamic is only on driver site, but can't when dynamic is on kernel site. Let's remove this code and move to MSI-X allocation feature by feature. If there is no more MSI-X for a feature, a feature is working with less MSI-X or it is turned off. There is a regression here. With MSI-X splitting user can run RDMA and eth even on system with not enough MSI-X. Now only eth will work. RDMA can be turned on by changing number of PF queues (lowering) and reprobe RDMA driver. Example: 72 CPU number, eth, RDMA and flow director (1 MSI-X), 1 MSI-X for OICR on PF, and 1 more for RDMA. Card is using 1 + 72 + 1 + 72 + 1 = 147. We set pf_msix_min = 2, pf_msix_max = 128 OICR: 1 eth: 72 flow director: 1 RDMA: 128 - 74 = 54 We can change number of queues on pf to 36 and do devlink reinit OICR: 1 eth: 36 RDMA: 73 flow director: 1 We can also (implemented in "ice: enable_rdma devlink param") turned RDMA off. OICR: 1 eth: 72 RDMA: 0 (turned off) flow director: 1 After this changes we have a static base vector for SRIOV (SIOV probably in the feature). Last patch from this series is simplifying managing VF MSI-X code based on static vector. Now changing queues using ethtool is also changing MSI-X. If there is enough MSI-X it is always one to one. When there is not enough there will be more queues than MSI-X. There is a lack of ability to set how many queues should be used per MSI-X. Maybe we should introduce another ethtool param for it? Sth like queues_per_vector? v5 --> v6: [5] * set default MSI-X max value based on needs instead of const define (patch 3) v4 --> v5: [4] * count combined queues in ethtool for case the vectors aren't mapped 1:1 to queues (patch 1) * change min_t to min where the casting isn't needed (and can hide problems) (patch 4) * load msix_max and msix_min value after devlink reload; it accidentally wasn't added after removing loading in probe path to mitigate error from devl_para_driverinit...() (patch 2) * add documentation in develink/ice for new parameters (patch 2) v3 --> v4: [3] * drop unnecessary text in devlink validation comments * assume that devl_param_driverinit...() shouldn't return error in normal execution path v2 --> v3: [2] * move flow director init before RDMA init * fix unrolling RDMA MSI-X allocation * add comment in commit message about lowering control RDMA MSI-X amount v1 --> v2: [1] * change permanent MSI-X cmode parameters to driverinit * remove locking during devlink parameter registration (it is now locked for whole init/deinit part) [5] https://lore.kernel.org/netdev/20241024121230.5861-1-michal.swiatkowski@linux.intel.com/T/#t [4] https://lore.kernel.org/netdev/20240930120402.3468-1-michal.swiatkowski@linux.intel.com/ [3] https://lore.kernel.org/netdev/20240808072016.10321-1-michal.swiatkowski@linux.intel.com/ [2] https://lore.kernel.org/netdev/20240801093115.8553-1-michal.swiatkowski@linux.intel.com/ [1] https://lore.kernel.org/netdev/20240213073509.77622-1-michal.swiatkowski@linux.intel.com/ Michal Swiatkowski (9): ice: count combined queues using Rx/Tx count ice: devlink PF MSI-X max and min parameter ice: remove splitting MSI-X between features ice: get rid of num_lan_msix field ice, irdma: move interrupts code to irdma ice: treat dyn_allowed only as suggestion ice: enable_rdma devlink param ice: simplify VF MSI-X managing ice: init flow director before RDMA Documentation/networking/devlink/ice.rst | 11 + drivers/infiniband/hw/irdma/hw.c | 2 - drivers/infiniband/hw/irdma/main.c | 46 ++- drivers/infiniband/hw/irdma/main.h | 3 + .../net/ethernet/intel/ice/devlink/devlink.c | 102 ++++++- drivers/net/ethernet/intel/ice/ice.h | 21 +- drivers/net/ethernet/intel/ice/ice_base.c | 10 +- drivers/net/ethernet/intel/ice/ice_ethtool.c | 9 +- drivers/net/ethernet/intel/ice/ice_idc.c | 64 +--- drivers/net/ethernet/intel/ice/ice_irq.c | 275 ++++++------------ drivers/net/ethernet/intel/ice/ice_irq.h | 13 +- drivers/net/ethernet/intel/ice/ice_lib.c | 35 ++- drivers/net/ethernet/intel/ice/ice_main.c | 6 +- drivers/net/ethernet/intel/ice/ice_sriov.c | 154 +--------- include/linux/net/intel/iidc.h | 2 + 15 files changed, 328 insertions(+), 425 deletions(-) Reviewed-by: Jacob Keller