From patchwork Sun Aug 27 13:13:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Madalin Bucur X-Patchwork-Id: 806245 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xgFvH3Xw6z9ryr for ; Sun, 27 Aug 2017 23:22:51 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xgFvH2cc9zDrpm for ; Sun, 27 Aug 2017 23:22:51 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0075.outbound.protection.outlook.com [104.47.40.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xgFj43rGjzDqgP for ; Sun, 27 Aug 2017 23:14:00 +1000 (AEST) Received: from BN6PR03CA0050.namprd03.prod.outlook.com (10.173.137.12) by BN6PR03MB3315.namprd03.prod.outlook.com (10.174.233.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9; Sun, 27 Aug 2017 13:13:54 +0000 Received: from BN1AFFO11FD034.protection.gbl (2a01:111:f400:7c10::121) by BN6PR03CA0050.outlook.office365.com (2603:10b6:404:4c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9 via Frontend Transport; Sun, 27 Aug 2017 13:13:54 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BN1AFFO11FD034.mail.protection.outlook.com (10.58.52.158) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1341.15 via Frontend Transport; Sun, 27 Aug 2017 13:13:54 +0000 Received: from fsr-fed2164-101.ea.freescale.net (fsr-fed2164-101.ea.freescale.net [10.171.73.197]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id v7RDDhJo028627; Sun, 27 Aug 2017 06:13:53 -0700 From: Madalin Bucur To: , Subject: [PATCH v4 6/7] Documentation: networking: add RSS information Date: Sun, 27 Aug 2017 16:13:42 +0300 Message-ID: <1503839623-3906-7-git-send-email-madalin.bucur@nxp.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1503839623-3906-1-git-send-email-madalin.bucur@nxp.com> References: <1503839623-3906-1-git-send-email-madalin.bucur@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131483132348215212; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(336005)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(189002)(199003)(86362001)(2906002)(8676002)(81166006)(48376002)(81156014)(8936002)(33646002)(356003)(305945005)(5660300001)(50466002)(5003940100001)(85426001)(50226002)(4326008)(498600001)(106466001)(104016004)(105606002)(626005)(47776003)(68736007)(53936002)(77096006)(2950100002)(189998001)(97736004)(36756003)(54906002)(3450700001)(50986999)(6666003)(76176999)(43066003); DIR:OUT; SFP:1101; SCL:1; SRVR:BN6PR03MB3315; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1AFFO11FD034; 1:039kDCrL2zOBYziq11e6KHRZ5bVueL9EZRLfjsbUhfbLs9K+kbHnwTXZtU7/7GN+7D+N39cdVYQemGzH5adrrBN3fE+RugijAVbPpfAUAPSK88Iy0G4DwB4I7Xz1uIRR MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: adbb6ce0-a53c-4e55-ce4b-08d4ed4d7806 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(300000503095)(300135400095)(2017052603199)(201703131430075)(201703131517081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:BN6PR03MB3315; X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB3315; 3:kuHXwc9qJTbbYWUalddbCamHMFOjZLFdu18vQD/8EwMVdew4ogTkuqsgrFgHCqD2S9oPYixGLIkSQItQIB8f6epngSjNchmqXjHMSulRqzJY+hFlaAKnd+IwHz3y8XlH91rdFVdIbRiCyy73Wh8Lg7hhUugK1HMNKRFh494JhhYhmkqy8azgC3CAaBVeCWu0CCyQHU+1sLMlP8XUUEXqQ4Ks7Rn2b1LREFQlqqwhE4l8RN6JhFsMB1k+9bRIyMs2xq5mkIvi2FjgTH4brefQ78N8BG/SXtzzuTelyLBU9IEyDwPX7biOQAnp13vfQIVcobHkEOdVunCO4M4XsUT+46hFRx2vatPOM2bvv0hMijQ=; 25:rXXWZDuoUrjxgUkNHB+dhKy9M+CoeotlHRvLizz9IRw7mS4Jm2PfFh7i/wLq5YHwTe1kWCca7v9NQ36n4NjeRDudLRiFQxD35c3XJfEUfjahRWI8Vv/mUbuTTrg0xqCd1ZwHQXqNEnMquMWBitVrky/wukOv9P3h46ThZiwHLe4X+BKgKwPIKBPBkYJjSoDM9pqlnVyZjSaneLzMWzYu766rrpvr/b62ikdRTCc05ZZ7xfYPPj/I+BrBqSdk2zTXP3vIh1KJDG+aw/heinO35hTkeQT3J29c0L5AcuDKqaEs3vy3u0F9DswASsNtOnsNJjXyir/VNd/+DNJXd7y0NQ== X-MS-TrafficTypeDiagnostic: BN6PR03MB3315: X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB3315; 31:Q2JuRBZ89OE1Ap/YESu3vC+b3PpdK05WpD12FGjTPSx79opoM3Uuhsq8rIag5LkDsF6g7exrZi9rHvDFIBDMm5xZ/JvvZZaSHnF3VceF6fKGqnY9InbnCAYljstPVH21PzwaV3hSnZ9XP/GeIpKw8mjKJ1mpNJvurwu3RO4bIxcaT6U0+cpfbrAPZozwuMFPt5TJpT8fZGWAVYKoMPx4Q4bN42GOIjRZt78/BIA6fsA=; 4:tC0jO4hTIWs8zDyYIinXGfb4Z7jJdEdXrVjy0GILRFxh2RkD4S9/F3pkl29vCFbrr1W+YUKRyf7sfIWQOfcYB2j+5WMlT+On//JZAhJxyZV+ph+4swvzUCptXh9RIpSi90R7kx49RiGNcQofExpvJYezHMY9WDKjTthQf9R+0zs/k0E3wFAFhWnMRqYcOJLct745PIUYC6/tqqDuLTzB+CtruV4PwIiiLipNSOg2/xuA8ARsdjO+hkp30aYB1XNmY5SEhqvHSf3h7BKvkL/OknplAhuzAGug1Z6sn+3C/3g39AwKvFDKL3qjc8gEhVvcGxzqB83CNFCu5I4bdc6DqMyHKQ0KnVvvgf/fyRnCVuf5EY7q9WxZhnm3ovlqJOMV X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(211171220733660); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6095135)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(3002001)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(6055026)(6096035)(20161123565025)(20161123563025)(20161123556025)(20161123561025)(20161123559100)(201703131430075)(201703131448075)(201703131433075)(201703161259150)(201703151042153)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN6PR03MB3315; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(400006)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN6PR03MB3315; X-Forefront-PRVS: 0412A98A59 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR03MB3315; 23:WuuwmaxVHkrl52n9r4OaOBj1Ju/kbuk2GBOLZ5GEr?= Qvhyvq6cqwoPYUfgRb0Th2V2Ku6qU2Rlg6/715eiZTSPxP6ty+Y9/FBdlxMWyWOvkvdvK4Vc+1QOt3RAec/018bgTtY8Te3BrddTsJZX2m/DVQOHgSINjJz3yJMFPMPkAT5wP3NSMegeGuaBzWuYScbBCItDrRepRT9Rw5ffflWTggll7hHcr9UUrUhibneKjcKEyllBDjBus6U6v5AAAcVEupjSEymt/9B7bB91GUb9xV7D5pvQCj8cK+17Xt8QSA46ACrTj5N7FsYatVFYHPpNO4wKlMIEOo6wRopk84EQAHbIPrxyrTfyX19QKoWxMrNudDGx5yT8ALD46+I+rLnoaYQMMgm5o86Kh6xcvVIp4LCZ0vcNvlCFh09M4ITHe+FUlqa6tpT+bJjBZFiz56H6zoRECsWGc4tpsz6/BCF79eABZrTXSdbZTcACXfHlYwSf6DKzEt93XbpQDHqcENbjFOfQHvpOPVhvBjMiLMEeSjO1t3rw9ScH1xb4aooRdxJLR9c/znUHVE7pe+/IW2jp6k/Z6JInFIBcnnvteswxAEsnshWzvqmcRYAxJuO/GIdFyJpI/NRI+pfXxSX2EVZF1zQmwJrIODI0Vb4g8pXWBCx3VLP12nABwgbRMNDtXy+pe1nxychq3Yd3UDnC6WUl0cqkS4HNwf1vCAJCSz1e022kMfZb9j5BCkzSYZGPTa865fZsDFlGENlZ4s9eA1deBbzEO4a7eULHZDwo5CjIpmX8hPj+N0Al0blOuISNkVPZKBUxRdzysLMCDuvBDID3wnZM1DFVn1YBo5iZOAmMj0t35NSAAcmd4AwR8GNNph35hvkmTdZNePu+Zz+TpNNsXjjwraMzX5CaUfka0BEBE7nnweDDJX4Z/RJT7FMhVH6tmQo8RbhwrjKAHc8PSHE2ejvoksAHaYBudppu+Fk0l7G9xVxTkFUHqtvMhs0oU7bTs0/b3HlLAQx+/lsUbV3JJcKEjduMJr70psgsSyaBufvN42BbhEe6lFryBqNim/ninP9EOGF5B6Y+NdIGS4FNrktiEjbJgKAKxYkWz9aCg== X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB3315; 6:njPCae+OUrCsvbZOUuxbP0xdsMmG+lgpcrLi9Rrh55k3BOjIwmPpFbBiuQM6mVKQIDQejNTF2/OWFcjpA4E0jMofPErqHh++fVeczY3vJ8tSrx4kB3fHcs+QnXDbAQDTZil4h1qtnMbfHGhKNaZgSXWk642tvgZ5E5W7EmtBHhqkztqrlnoXuct+RWufA1OIeqiQGqmUMCEgWLteNVpP41r8JTzWEIkGPi55Y7ZZYXacI/zHh2ow33em8am66nkdm3jOmvcGFXwhXSxIeD4KxAdJYHl1zncTnlFNVra9bzy6wImzLY2wrqFhwHkwrMuxLzJgxQZNLpb0aXkoC/Vdrg==; 5:ZxpnzOsBqkUMsESi8f8Yw+6qXwLbAQalBEgAGEy7RSCriigdjqsCbOvRpThu6vHCZp1DSEEqZvLL+0hhPf4ghuoC+K85Nf5UJlWaXlBQqCUxcoRKr/R9BQa1B0/JYcmQLAE3c+brT7xsaN4xlpXoMg==; 24:r7E9AvKegyqxuTa6tMkLg7HFheX8bDefTC/MBbvGj0QdPZ0iosZZLd78HILsburFgVRlIMPT0AT1OxIXCLaTC/QuDuteCxMIo4KxkD9O2cI=; 7:X8IoXTwMq7hksAttkkspQRH4n1D/tnO+7+dKduVAzyFTpeATD5uohLyt0M2dbKO09+JKOS5bvX6lQ3sqYsSXbbq8Efk2+4jXgirp9468BCOdX79dSpArh6ejMN1YQcpBdyrXQ2Hvg238wb3q3+r+KUzA6gpS+UCUDP+LL6DSbwV+4V03d4f6lH4Q7bQPWE+E7K/Hl3XHEDmz52gHEHXmnT8YR+yjyt8LOioAHr+zfJ4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2017 13:13:54.6499 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB3315 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: madalin.bucur@nxp.com Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Signed-off-by: Madalin Bucur --- Documentation/networking/dpaa.txt | 68 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/Documentation/networking/dpaa.txt b/Documentation/networking/dpaa.txt index 76e016d..f88194f 100644 --- a/Documentation/networking/dpaa.txt +++ b/Documentation/networking/dpaa.txt @@ -13,6 +13,7 @@ Contents - Configuring DPAA Ethernet in your kernel - DPAA Ethernet Frame Processing - DPAA Ethernet Features + - DPAA IRQ Affinity and Receive Side Scaling - Debugging DPAA Ethernet Overview @@ -147,7 +148,10 @@ gradually. The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx checksum offload feature is enabled by default and cannot be controlled through -ethtool. +ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS +provides a big performance boost for the forwarding scenarios, allowing +different traffic flows received by one interface to be processed by different +CPUs in parallel. The driver has support for multiple prioritized Tx traffic classes. Priorities range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with @@ -166,6 +170,68 @@ classes as follows: tc qdisc add dev root handle 1: \ mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1 +DPAA IRQ Affinity and Receive Side Scaling +========================================== + +Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation +queues is seen by the CPU as ingress traffic on a certain portal. +The DPAA QMan portal interrupts are affined each to a certain CPU. +The same portal interrupt services all the QMan portal consumers. + +By default the DPAA Ethernet driver enables RSS, making use of the +DPAA FMan Parser and Keygen blocks to distribute traffic on 128 +hardware frame queues using a hash on IP v4/v6 source and destination +and L4 source and destination ports, in present in the received frame. +When RSS is disabled, all traffic received by a certain interface is +received on the default Rx frame queue. The default DPAA Rx frame +queues are configured to put the received traffic into a pool channel +that allows any available CPU portal to dequeue the ingress traffic. +The default frame queues have the HOLDACTIVE option set, ensuring that +traffic bursts from a certain queue are serviced by the same CPU. +This ensures a very low rate of frame reordering. A drawback of this +is that only one CPU at a time can service the traffic received by a +certain interface when RSS is not enabled. + +To implement RSS, the DPAA Ethernet driver allocates an extra set of +128 Rx frame queues that are configured to dedicated channels, in a +round-robin manner. The mapping of the frame queues to CPUs is now +hardcoded, there is no indirection table to move traffic for a certain +FQ (hash result) to another CPU. The ingress traffic arriving on one +of these frame queues will arrive at the same portal and will always +be processed by the same CPU. This ensures intra-flow order preservation +and workload distribution for multiple traffic flows. + +RSS can be turned off for a certain interface using ethtool, i.e. + + # ethtool -N fm1-mac9 rx-flow-hash tcp4 "" + +To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6: + + # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn + +There is no independent control for individual protocols, any command +run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is +going to control the rx-flow-hashing for all protocols on that interface. + +Besides using the FMan Keygen computed hash for spreading traffic on the +128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when +the NETIF_F_RXHASH feature is on (active by default). This can be turned +on or off through ethtool, i.e.: + + # ethtool -K fm1-mac9 rx-hashing off + # ethtool -k fm1-mac9 | grep hash + receive-hashing: off + # ethtool -K fm1-mac9 rx-hashing on + Actual changes: + receive-hashing: on + # ethtool -k fm1-mac9 | grep hash + receive-hashing: on + +Please note that Rx hashing depends upon the rx-flow-hashing being on +for that interface - turning off rx-flow-hashing will also disable the +rx-hashing (without ethtool reporting it as off as that depends on the +NETIF_F_RXHASH feature flag). + Debugging =========