From patchwork Fri Aug 18 08:56:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Madalin Bucur X-Patchwork-Id: 803107 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xYccr18g2z9sR8 for ; Fri, 18 Aug 2017 19:05:48 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xYccr09W1zDrRb for ; Fri, 18 Aug 2017 19:05:48 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0046.outbound.protection.outlook.com [104.47.42.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xYcQM3P2GzDrLt for ; Fri, 18 Aug 2017 18:56:43 +1000 (AEST) Received: from MWHPR03CA0002.namprd03.prod.outlook.com (10.175.133.140) by DM5PR03MB3321.namprd03.prod.outlook.com (10.174.241.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1362.18; Fri, 18 Aug 2017 08:56:39 +0000 Received: from BL2FFO11FD033.protection.gbl (2a01:111:f400:7c09::184) by MWHPR03CA0002.outlook.office365.com (2603:10b6:300:117::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1362.18 via Frontend Transport; Fri, 18 Aug 2017 08:56:39 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BL2FFO11FD033.mail.protection.outlook.com (10.173.161.129) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1341.15 via Frontend Transport; Fri, 18 Aug 2017 08:56:38 +0000 Received: from fsr-fed2164-101.ea.freescale.net (fsr-fed2164-101.ea.freescale.net [10.171.73.197]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id v7I8uSXY011416; Fri, 18 Aug 2017 01:56:37 -0700 From: Madalin Bucur To: , Subject: [PATCH 5/6] Documentation: networking: add RSS information Date: Fri, 18 Aug 2017 11:56:27 +0300 Message-ID: <1503046588-24349-6-git-send-email-madalin.bucur@nxp.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1503046588-24349-1-git-send-email-madalin.bucur@nxp.com> References: <1503046588-24349-1-git-send-email-madalin.bucur@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131475201989331343; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(336005)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(199003)(189002)(6666003)(47776003)(2950100002)(36756003)(97736004)(77096006)(5660300001)(54906002)(189998001)(104016004)(85426001)(86362001)(5003940100001)(498600001)(50466002)(4326008)(356003)(43066003)(8936002)(305945005)(81166006)(8676002)(50226002)(81156014)(48376002)(626005)(50986999)(76176999)(53936002)(3450700001)(2906002)(105606002)(106466001)(33646002)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR03MB3321; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD033; 1:EMaMRccSmUfVAEiUUhEHY6+eB/HB0N5YCzisjJqgm+WkDn/dCnqTgKztLm/1tdTof4L/NSOUA8rmGvnWT5WuNNwfb7KYuDDOC1pe30UJxcfVSa9tjBfm+M/3B5w3wHg2 MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0ac65753-4614-4224-682f-08d4e61709cd X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(300000503095)(300135400095)(2017052603031)(201703131430075)(201703131517081)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:DM5PR03MB3321; X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3321; 3:ji5QW1JknxagvIhIDEOlqv+aZpN7Uw7H44Hear35naDc3/s9vAwk68qrsgnx2nwZj3EUIKEzcjhZGLeus0qllx1fzv+UF3Ri38r7vG00poS/64wd00lYiSy8Jk8sOcWY8b9HpPeIWDL9sM0oGGy1HtB9OTYLtt/6sYXMlRQ2Vu/L6zCCdAgek+ktFyZgZa5XtaBeKrwiffWco9G9WYlXxu2MwL/NXBPO29rQvqy93DVFTyBKCbjK/bzmGlrV6Ejq99UuH/B08rzAvn+hUByH7ogCWqX8UaWnqlg9Z65XSX4YfB6wC5r8Sl4xufPscMStS7/VktNFasdEIE4D7t6uu0UaIxvg2jR/Knf3w7yivkE=; 25:LQ6qQFkLu6VZfbxA/I9DFWdOzu1q3NmbVCLFEr8DjK8+3ifew5/eAllxJXRqOoL76Ahh939Crf84lL1Okju0Nfpa3sBXiMjN08/gdsl2YEVXCgjdZ4WOk+XrlDKvLMTNfzWXQOcySWm4wjEImXFU5FaU+qO1a5lBAd9HMDQY+9mSp/NIv+y/dsI2PfurLpuE2kVfrDwSFYq8YCmg5fjj0TLJv7uGG+ZTzuWuRIhIfv8yUB9h/LX29rR4cQgPxHb2GjPMS5fo/sNG5SUf66Dg0A+PKbv7DuJUIvAYvmfLvFXcyHVm4ddf+xgbmbuBFinMWE9xwkwFZyVIi9aS92G/BA== X-MS-TrafficTypeDiagnostic: DM5PR03MB3321: X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3321; 31:2hMO6D8N+t5FUEXmZ+sbW/FebxLlmloAzE097YGL7E3ORT7PFR00fA2TRrt+SKcx/AiL3NBi/RNgZOISuuWpNFSzoK9XW50N8FfnXBM26WzdEv/b3fTqjxoc365Ig8Xy5WX0Obg/Tfz03X0TmCuY/QuFDg5p69FmSY2oRoilA8vm7vvfNoJ4MilCdYFIF+ymIu+m9dF92NZ5Q5VIqP3UQYhf8oZdyTtdbnwxjq8a0I0=; 4:14qnRmyhKeH/qZs6tSrJDz+ZdM6VTO/hSYixvmxfauyTNMuFLf4SOzUCYBfjjRhnSKNzIv32+1BFYrOCW5aR3GRPVfopp2g+jOl6Hqde4k2lfi4gK336pcOUqJkXODBq9q+PPgdBSuGWBNu/O2t9T+Dq58z8OOsqKwNyG95HAcZG0i8XPfCTljbI8y4hwrsFVWE8zjMrbY38WSXvQ6gKFdXq6Sp9asHqe2T/MmYs8HLyIep7mW+cmyWgjzc4Dev5HfZq1lxoD3kHOociinYFnQZQZLPGMDWxAEQZVPlQqjAtmLmuCxPSfGguRM1X2Pt7lrWQ2qaErra9O3ch3m4eKU9TbCVr+npsb97bm6qOWUZ1EKi+HMiZRnXh0svL+ETJ X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(275809806118684)(211171220733660); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6095135)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(3002001)(6055026)(6096035)(20161123563025)(201703131430075)(201703131433075)(201703131441075)(201703131448075)(201703161259150)(20161123559100)(20161123561025)(20161123565025)(20161123556025)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR03MB3321; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(400006)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR03MB3321; X-Forefront-PRVS: 040359335D X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR03MB3321; 23:oGbMK4GIykGeUPWsr9lezTsluZ4uJao73/bqq4wrL?= =?us-ascii?Q?UwTyS/PcpFuFhEKNr3kduq4Z0QrGKIDQBVulGXQ+ou0VIf73UpXtpEkd3gPm?= =?us-ascii?Q?35xBwE3AyxIeITtNEt4ddhDxmiP1xnMkPg7hsdMdWge2Mc4SQZ/rB8QjCdbo?= =?us-ascii?Q?fExu3maLEZG20F1WE8uDPpsl6uhxfVzd0NUSg1vukVvC00mxbT5jdrdppIGv?= =?us-ascii?Q?FijV2/xhPNSxNVwbvHL5J7c9MaliCvUwGCk+Xp4HRMhaa6TU6HJOLBJ3vp/y?= =?us-ascii?Q?kLNVr2mGvAtThLqoVmuMpK6ywVow0cuDA45dPPS9roxASKocNIATi6b10hdd?= =?us-ascii?Q?wQoFheFvDBdsO1iL7Nr8/igW7B9zfJAQxbCdlErTt8EttiWYjcHNYz7dtXba?= =?us-ascii?Q?fRiLZqQ9BNbOfAZUr72zwYk+NPynAUR9wCIseAoMFz9fjy+uevxEYTb9yrao?= =?us-ascii?Q?uJyo3x8bMg7UxSxJJwb0S+KFpVmAcfjqM1552tqFDxMwtlZzCw9GQ24pvNEs?= =?us-ascii?Q?cxvyUeun3Wb6A61+chdKWnWNrrku7L1M/Rds/B4zYl6H5CZbQAv3BMthjRGy?= =?us-ascii?Q?p6V/ojsq/L3rcmVkvYs4XBdczJPNN/TLu+hPMMS1fgyLYHqZu5Cw7YO9tfqq?= =?us-ascii?Q?y5T6sezywo/8dwXXy2mBt+OfrLMtFiQCuzhb2YxZmPKk9KZLk6AXYWgkW0Ke?= =?us-ascii?Q?2pOdX/un4XvojFD3lP48fgxNZ/aHY7JHJevIPMRkkUurI8uarA8yFtDRH6gy?= =?us-ascii?Q?6ej8KFaAJZYj4o7AScRfwuV5vgeJz5iRmDsGP5dmTF9zXbn8noLhLKlX8FCG?= =?us-ascii?Q?QQHrt1cxMCRasQdEDAJJ8NOm25wjpBDhx/yJ4pghTv+VP0lAs8H98Dp9QwmT?= =?us-ascii?Q?t+dQgIWbFAq8xxkcVbHynTNWCDvNq5+RqRwac45uhNkLdUm02CU3mllKUuRb?= =?us-ascii?Q?GN2oEU7YiVfgIfRVJhVsDK/bnLUyn4IXM+8ta0w4/RR5g0bh1/DW/MxT2GvJ?= =?us-ascii?Q?u1COn5uzOLtUifFYNHfBRZN6i2PsVDa+nDrLyp2hm9HK4QJJyfalJ7FRecMr?= =?us-ascii?Q?7ydIlcjWOyEfBwPh0iEkmIIkxywlGu7cjDlhA4mxtqP72+yFw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3321; 6:UxHwG4LNgpxdy4qLdQ0OV9hDd8iM8R9IFUixQ1Lq8+41NMYPDWe6dTW9zz/fmGg6ZP1Es2G3EEdrrQkfCBYVQis6eXIjfIFMT+CLIe48wbM2ajszcZ9vunG1XyyR3Elf2tEBbpo144xRuXnKawZo0mWJToNbyxLVu93Pq5WYZtkn4saCuHi3KpVzNUE5rlogtnXQ7w0PgsvXqmqVkFH4XoPZ91Nmlcs+gMfRwaap+wS7+FPVJBsJFbkP4IR3KHe75jtmM4QuJcpnu440vcJNSZiX87D6ti3VOrPVg2q9Nry0uKp0zkg9PucaKZg1MdKoT7GwpqRd8CKQrPGEmNO02A==; 5:fIr5sq5mNFi+9XcvZbHb09A7p5fZeUfcoA6kDsjHJl8rrSAT+BTLucUpEi7F4DntLDmZtJtzyA5UpHpNGhU9/hvohXljt1v6jtjPZ/aRI3ynmlj80G5H2YKRCo41qNfqtxvBCLgbjrJ43XOnTRb2+A==; 24:R2ijgqsvjR9egv5QVCRTwCUFnZeINqmpN5r3asa2bsiDImynA7ql8dOMFYYECbJsQJ/FnuhpU7uunkXRfuY6xKfzSbsPgkP5cuoDs0w6mBQ=; 7:2UzRIi7IBRak5AHUJ4DWXU5LHUAbPpZ78JlX5vxTP6wGV6G4BBK+OmD32GSSH6MTehAahQ1apWDdKO0doGB9lCe4ndlI/Zm3VREqEB+vlQW2lm9C9wSJ+7tBxKpZ+C/saj0xxwQBxvcjypAwWTrx456vQSfoq2Wz0QWugO0rGhspurHi+KVMZQG25qVhWo4k2EQeUa3PbAvAQ+24dRDFJw3VjCSIiWn3aMUvfZPAz0o= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2017 08:56:38.7615 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB3321 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: madalin.bucur@nxp.com Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Signed-off-by: Madalin Bucur --- Documentation/networking/dpaa.txt | 68 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/Documentation/networking/dpaa.txt b/Documentation/networking/dpaa.txt index 76e016d..f88194f 100644 --- a/Documentation/networking/dpaa.txt +++ b/Documentation/networking/dpaa.txt @@ -13,6 +13,7 @@ Contents - Configuring DPAA Ethernet in your kernel - DPAA Ethernet Frame Processing - DPAA Ethernet Features + - DPAA IRQ Affinity and Receive Side Scaling - Debugging DPAA Ethernet Overview @@ -147,7 +148,10 @@ gradually. The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx checksum offload feature is enabled by default and cannot be controlled through -ethtool. +ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS +provides a big performance boost for the forwarding scenarios, allowing +different traffic flows received by one interface to be processed by different +CPUs in parallel. The driver has support for multiple prioritized Tx traffic classes. Priorities range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with @@ -166,6 +170,68 @@ classes as follows: tc qdisc add dev root handle 1: \ mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1 +DPAA IRQ Affinity and Receive Side Scaling +========================================== + +Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation +queues is seen by the CPU as ingress traffic on a certain portal. +The DPAA QMan portal interrupts are affined each to a certain CPU. +The same portal interrupt services all the QMan portal consumers. + +By default the DPAA Ethernet driver enables RSS, making use of the +DPAA FMan Parser and Keygen blocks to distribute traffic on 128 +hardware frame queues using a hash on IP v4/v6 source and destination +and L4 source and destination ports, in present in the received frame. +When RSS is disabled, all traffic received by a certain interface is +received on the default Rx frame queue. The default DPAA Rx frame +queues are configured to put the received traffic into a pool channel +that allows any available CPU portal to dequeue the ingress traffic. +The default frame queues have the HOLDACTIVE option set, ensuring that +traffic bursts from a certain queue are serviced by the same CPU. +This ensures a very low rate of frame reordering. A drawback of this +is that only one CPU at a time can service the traffic received by a +certain interface when RSS is not enabled. + +To implement RSS, the DPAA Ethernet driver allocates an extra set of +128 Rx frame queues that are configured to dedicated channels, in a +round-robin manner. The mapping of the frame queues to CPUs is now +hardcoded, there is no indirection table to move traffic for a certain +FQ (hash result) to another CPU. The ingress traffic arriving on one +of these frame queues will arrive at the same portal and will always +be processed by the same CPU. This ensures intra-flow order preservation +and workload distribution for multiple traffic flows. + +RSS can be turned off for a certain interface using ethtool, i.e. + + # ethtool -N fm1-mac9 rx-flow-hash tcp4 "" + +To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6: + + # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn + +There is no independent control for individual protocols, any command +run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is +going to control the rx-flow-hashing for all protocols on that interface. + +Besides using the FMan Keygen computed hash for spreading traffic on the +128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when +the NETIF_F_RXHASH feature is on (active by default). This can be turned +on or off through ethtool, i.e.: + + # ethtool -K fm1-mac9 rx-hashing off + # ethtool -k fm1-mac9 | grep hash + receive-hashing: off + # ethtool -K fm1-mac9 rx-hashing on + Actual changes: + receive-hashing: on + # ethtool -k fm1-mac9 | grep hash + receive-hashing: on + +Please note that Rx hashing depends upon the rx-flow-hashing being on +for that interface - turning off rx-flow-hashing will also disable the +rx-hashing (without ethtool reporting it as off as that depends on the +NETIF_F_RXHASH feature flag). + Debugging =========