From patchwork Thu Aug 24 07:28:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Madalin Bucur X-Patchwork-Id: 805290 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xdGQM44mJz9sRm for ; Thu, 24 Aug 2017 17:39:23 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xdGQM3FkwzDrcl for ; Thu, 24 Aug 2017 17:39:23 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0079.outbound.protection.outlook.com [104.47.32.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xdGB40zvFzDqXv for ; Thu, 24 Aug 2017 17:28:44 +1000 (AEST) Received: from BN6PR03CA0083.namprd03.prod.outlook.com (10.164.122.149) by BN3PR03MB2260.namprd03.prod.outlook.com (10.166.74.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9; Thu, 24 Aug 2017 07:28:40 +0000 Received: from BY2FFO11FD007.protection.gbl (2a01:111:f400:7c0c::194) by BN6PR03CA0083.outlook.office365.com (2603:10b6:405:6f::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9 via Frontend Transport; Thu, 24 Aug 2017 07:28:40 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BY2FFO11FD007.mail.protection.outlook.com (10.1.14.128) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1341.15 via Frontend Transport; Thu, 24 Aug 2017 07:28:39 +0000 Received: from fsr-fed2164-101.ea.freescale.net (fsr-fed2164-101.ea.freescale.net [10.171.73.197]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id v7O7STqg005094; Thu, 24 Aug 2017 00:28:38 -0700 From: Madalin Bucur To: , Subject: [PATCH v3 6/7] Documentation: networking: add RSS information Date: Thu, 24 Aug 2017 10:28:27 +0300 Message-ID: <1503559708-13405-7-git-send-email-madalin.bucur@nxp.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1503559708-13405-1-git-send-email-madalin.bucur@nxp.com> References: <1503559708-13405-1-git-send-email-madalin.bucur@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131480333200577488; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(336005)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(189002)(199003)(50466002)(86362001)(48376002)(104016004)(2950100002)(6666003)(53936002)(43066003)(47776003)(106466001)(33646002)(50986999)(76176999)(105606002)(50226002)(498600001)(5660300001)(54906002)(97736004)(81156014)(8676002)(81166006)(77096006)(85426001)(8936002)(305945005)(2906002)(68736007)(626005)(4326008)(5003940100001)(3450700001)(189998001)(356003)(36756003); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR03MB2260; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD007; 1:Bkk9ujK1zrOQzLqsZteIvSJHbPhKKdzUAbax7g56Q3YXelIjtL1GORaYFldBXgPXNK2AkVdgHoyQGLQP5gCGBxxMtaUETJbLBJ9deSPRc7F9caeq48Pq37wEKVhya8y/ MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0fb34049-7e9a-4cf9-726f-08d4eac1bdd3 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(300000503095)(300135400095)(2017052603196)(201703131430075)(201703131517081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:BN3PR03MB2260; X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB2260; 3:mPsaczV0kWjjusmz7FQxtdjj3Yesvd47A/u/Flbk/MXztjhPTpiNhi276ZJaIEj4DUx2y9gDvDii43yDjmxfXtIq3BRN0kh7YCqewyi5AHQiwMrLzApIWWB7V6iGDZJUw8s/QfvBf6t0YiEAnfGF1DoJsifwY7QK0Lq1AswG1GFTKvxEmTrNUXj1F2FEfRNlVGdMR9IILEWXK5vlY9mHvkSOSHMaVJda2RE5VeCbxcA/seaxROk7HHgQ9TjpqxNhApwFaxYZl52gGRutGvYn4xEzh40v/o7LqROuokUwmRS7STrAPMLXMNVyfJWCWHsgHv2pw5ssmAFYSGva3DE4+Db1UfZQnCBAEFp3tCA8fnU=; 25:lZGIJoHh+k6S7x5E1oBlpKQOmmp5evyncEyhvg2C8SHSAroOr6S8KyRZC84d6kp57BXhCZnrSN8cVu3atyfoUMHlgm7ZqF45c0SRtFLoNVnsUJzvPLfAvYtMWXkMkqhPtCW31x511sQBxnbS6476SmTa/2t1vOBjWcBKURUbu0GoKrGerM2zgiwaRMr5F9AYKd8/fQE9p0M8r8DAnaW1kNYinTcORIvTmnd2DpwmW22yfkck5be/pQctBOWzUoJoEU+0QVARUTd5gW2H6L4mPQBkPBhj+B06HeCN4rTj3OtikjA6h2T6oRwd2ppmevxLXDL1WtDnlDpYMeagsn8rVw== X-MS-TrafficTypeDiagnostic: BN3PR03MB2260: X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB2260; 31:FAbV310pAtpzbbao7UYBqzrxuHRi8cPGi4lwOZFF2iKv3fX/5uqQy3uMj1APbvlQFp+2knLCPWAZhXcjhZcnvAuTxpUXtAeu/+4xMP1Bdc1Mrgvi95pINwfmWACDugnQmrn6LnXSu13oWySAephlDguDKhJrYjlYNocBi7R3wFZZua3CxQV3ec9DBrbPRNrNqoX/Y9Gs+PHxeEaqlSXHrl7Pj1PXmbwoHdMIf/5R+/g=; 4:8WBQb7J3gshkJi4qa5rNOuxlIHBNYyq/BSsUOiu9tOGf62fKBwkdpIyVQJPhPV9CL0Bju2frExgylFOGCdDVvt9ok8LeXc3V4CYEYJtKVwVcIZmnondPR6jqHURhiIAZTp5eu5z3AwU2CJtbMLC+DzApoLNqfWIblg8npXX1Uy3q0eJW2CitU17jNy2JdnHnQnPSY3vZrUl6RnGJumszNX5+URrr9bRsye4cUPBRq+RnkcYXJDzDu+3uioOQimNwy/fMrRuyDHXNZ6wGJ+HREJi3Uy/mhMJA7vn6r44fjUBkNhTVnKGSN57TUP0x6LF50IfZsKMxQ3P0coayr3BK8id6IYE0pgZWybBp20DRtAMBr5ibBpdRYMZ7MZs9wcdW X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(211171220733660); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6095135)(601004)(2401047)(5005006)(8121501046)(13016025)(13018025)(100000703101)(100105400095)(3002001)(10201501046)(93006095)(93001095)(6055026)(6096035)(20161123559100)(201703131430075)(201703131441075)(201703131448075)(201703131433075)(201703161259150)(20161123561025)(20161123565025)(20161123563025)(20161123556025)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN3PR03MB2260; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(400006)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN3PR03MB2260; X-Forefront-PRVS: 04097B7F7F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR03MB2260; 23:U/OOK5Nreq9XCPfNHUd4J6P9ps6ma0UyU4Vfwq/9E?= =?us-ascii?Q?S+FQTshj+u9v7N85EP1e5zuc1szWlsJ+4ypVF6Ae2cjdVVWQbdN57eUh5JWp?= =?us-ascii?Q?v0fFZtDDKXOJ2jwsuSPSjAi4AO1J1kHDRA9DvajMsyuaGpMsPIYIhi1BXvAU?= =?us-ascii?Q?mP9BKHP3K+kkhohR1gsgEohRqquH8oLC4Lv0zpKtjGMi+BXRwRWOXEn3bOOK?= =?us-ascii?Q?sMEVxSirLUsSLRsDWQnbOA4o4Ui/G0vYGNBy5v0Sf+24/CQd7JvGqTTx3AF7?= =?us-ascii?Q?AcIIpjROABEmYpxNfzZOhlsRy71Nyv/XXuqcT1leP4Dz9ULNdNDQk7mNcB1W?= =?us-ascii?Q?3kMkusvpjheFJw3HT6CwHwlwrqX+KOs9k30WM6n+HdB/N7BvrIWJNPS/kIqh?= =?us-ascii?Q?ZvfyIxHnW5oSe6KCwlEro5Qg2E849GrmIMJ7OeAjymqiChe6MgS6ytbMNPmc?= =?us-ascii?Q?3Jpue1bQsWYtF2UyE0ZXFqnExGP8CVF2HJ47YnI7ocVHUebpEHNHT5wZ6ii0?= =?us-ascii?Q?cnWTcHTzLVwwDpjMMJOg91m0KgyNLVzlAqhj3mVbVTgKvoIilaiU486JkrIL?= =?us-ascii?Q?lyhBzPk0cOXWv+DBiYdtmjUq4a23beZt8IoVOfSzG0GEWyQYHyR25uu7tfr6?= =?us-ascii?Q?FtdkwYDTFtJvMJIr5+YKDpVlre7pLnH/ktJvmeAgninv0BpnXvHuZLkvsU/l?= =?us-ascii?Q?YaYETX77Gwc1XgaJNG7wT0LXBYQp8F1s+tmqDm/pFvmIKkYbdRhz+oAXQER5?= =?us-ascii?Q?Nc9TEkM2qW5OhagD+ddDYtC4ucU/UnPkHun9PagZ0Y2jE3BzKUxPnL2m0DjK?= =?us-ascii?Q?1si0wXBmhd0IGJrUf5E/cgGxCRJ/ySg6Mt5RRvgDVbaTsygRLJ+3zL1DWycc?= =?us-ascii?Q?/T255FTVwsvprAs038+Op+G0vRqQLfth1n+EjoWSzgMqHqQXJmbb4R8CGvqB?= =?us-ascii?Q?6y7YQOlxS9xzIkilU901IlJajr4pAnilf9rQQ/VSomYL5Ivpnbt2NaD0Z1lg?= =?us-ascii?Q?g6NtKT337rjwASIXYVXFPKEtkJx+3afjAbk7P/OZ0MZLB7XEGGcaLxvWDCIg?= =?us-ascii?Q?rq3LvHGXedqZXonJRmJD5qW2jSryOyxaagemor4ebH25qh4Jw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR03MB2260; 6:2RbawEHp6E9BI9u1XTtdLeCszeUyaboB2Br7kG9kN5Tv9BB5qQIUCipTNaRWPhUoBZMM8RUVhsmZj+EnAawwtXoUQ/hGhwHiidVH5hzq/vvGJYuQ7iELBWsXUiZXu7HuB4H+NTqOcYD7NUxzcjJu9NiVpqDrPBqlOPH4tPW91IVr77GIK4C9csYre8a3UJaO6369OtIpi9iao+SLcBhwW1lelB6FRpsrtD4Kq8U9nnyJWZ3HpUFD4lKsAPWm/BpXXfxcfc1MpX+clHMzj4vuessUeBhElIGbdGq7Lgue3hvAkr/kIxc4UrlvfeqOHPAz/8hCDTikIkP7qyzvQZz67Q==; 5:am4TZQqubDErF22XofCl3Ml4f61yhvsWxHXtGmqPW3nBbOkWohvIUqu+AEJ4Sak59y3YKdv3G4E4I5bDBKKvtNsxEgsA8pKDtftZYrE/GuPSO54Jg9H10cYc8OwtjqF3OSqyXEwvDtDRDBN4sko+fA==; 24:Hl3cZYYiWx/PFal60+bZLIaiCUWNTL9b5+gdJy8oiROnGEYDxAD/jbdRG/YsamLXmLzRfyOYOW5YHDGKylhr2SnaynNFt23S4o/1k7hG8LM=; 7:JaWnWDSqjr2IaAtJgQq+QWl+yycE56uiE7yp+zqK7ax/+UHrZSn9nQTHbkYhH4Pha/QTDld0wnFz9gxZ6bGkalUwOvai23UPsJkpF7Y/jIGbqfaJK+Q6aaNrpWBPOUbfiGeh0vtGAlOkfWEE1UxdidDo4cs0UhJhBeyy1loXTS0E1jNr0FfYS783sbtqO+Ak/AZmS0FIgx2KtxbmxcVig/dpy3fqEVEJE39HGsJ1VSI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Aug 2017 07:28:39.8549 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR03MB2260 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: madalin.bucur@nxp.com Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Signed-off-by: Madalin Bucur --- Documentation/networking/dpaa.txt | 68 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/Documentation/networking/dpaa.txt b/Documentation/networking/dpaa.txt index 76e016d..f88194f 100644 --- a/Documentation/networking/dpaa.txt +++ b/Documentation/networking/dpaa.txt @@ -13,6 +13,7 @@ Contents - Configuring DPAA Ethernet in your kernel - DPAA Ethernet Frame Processing - DPAA Ethernet Features + - DPAA IRQ Affinity and Receive Side Scaling - Debugging DPAA Ethernet Overview @@ -147,7 +148,10 @@ gradually. The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx checksum offload feature is enabled by default and cannot be controlled through -ethtool. +ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS +provides a big performance boost for the forwarding scenarios, allowing +different traffic flows received by one interface to be processed by different +CPUs in parallel. The driver has support for multiple prioritized Tx traffic classes. Priorities range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with @@ -166,6 +170,68 @@ classes as follows: tc qdisc add dev root handle 1: \ mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1 +DPAA IRQ Affinity and Receive Side Scaling +========================================== + +Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation +queues is seen by the CPU as ingress traffic on a certain portal. +The DPAA QMan portal interrupts are affined each to a certain CPU. +The same portal interrupt services all the QMan portal consumers. + +By default the DPAA Ethernet driver enables RSS, making use of the +DPAA FMan Parser and Keygen blocks to distribute traffic on 128 +hardware frame queues using a hash on IP v4/v6 source and destination +and L4 source and destination ports, in present in the received frame. +When RSS is disabled, all traffic received by a certain interface is +received on the default Rx frame queue. The default DPAA Rx frame +queues are configured to put the received traffic into a pool channel +that allows any available CPU portal to dequeue the ingress traffic. +The default frame queues have the HOLDACTIVE option set, ensuring that +traffic bursts from a certain queue are serviced by the same CPU. +This ensures a very low rate of frame reordering. A drawback of this +is that only one CPU at a time can service the traffic received by a +certain interface when RSS is not enabled. + +To implement RSS, the DPAA Ethernet driver allocates an extra set of +128 Rx frame queues that are configured to dedicated channels, in a +round-robin manner. The mapping of the frame queues to CPUs is now +hardcoded, there is no indirection table to move traffic for a certain +FQ (hash result) to another CPU. The ingress traffic arriving on one +of these frame queues will arrive at the same portal and will always +be processed by the same CPU. This ensures intra-flow order preservation +and workload distribution for multiple traffic flows. + +RSS can be turned off for a certain interface using ethtool, i.e. + + # ethtool -N fm1-mac9 rx-flow-hash tcp4 "" + +To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6: + + # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn + +There is no independent control for individual protocols, any command +run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is +going to control the rx-flow-hashing for all protocols on that interface. + +Besides using the FMan Keygen computed hash for spreading traffic on the +128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when +the NETIF_F_RXHASH feature is on (active by default). This can be turned +on or off through ethtool, i.e.: + + # ethtool -K fm1-mac9 rx-hashing off + # ethtool -k fm1-mac9 | grep hash + receive-hashing: off + # ethtool -K fm1-mac9 rx-hashing on + Actual changes: + receive-hashing: on + # ethtool -k fm1-mac9 | grep hash + receive-hashing: on + +Please note that Rx hashing depends upon the rx-flow-hashing being on +for that interface - turning off rx-flow-hashing will also disable the +rx-hashing (without ethtool reporting it as off as that depends on the +NETIF_F_RXHASH feature flag). + Debugging =========