From patchwork Mon Aug 13 18:43:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shannon Nelson X-Patchwork-Id: 957190 Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.b="UgIYoOjd"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 41q4Q36hRXz9s7c for ; Tue, 14 Aug 2018 04:44:11 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730415AbeHMV1Y (ORCPT ); Mon, 13 Aug 2018 17:27:24 -0400 Received: from aserp2120.oracle.com ([141.146.126.78]:49584 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729525AbeHMV1Y (ORCPT ); Mon, 13 Aug 2018 17:27:24 -0400 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w7DIefQq038579; Mon, 13 Aug 2018 18:43:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id; s=corp-2018-07-02; bh=UEB7SHkzXd6QoReef5yNnsNp/TQthXnrYeZyKrReljE=; b=UgIYoOjd9XZz3H4KT8KwF5X6uCon7tyVJPEkiw17dOGevJH5Q9WIpwy1gtYAPf8L7a4n 0Fiqn36fSKcID7P18F5CGoc/bMwFsk4m7NFReoDR/Pve62Jg40ItHVr+bTr9ET8rZ9pT UwltHsrAOZNEehYxB1/fyyo+N9LBz8dHSJCSuT9wzMxSlJMfWepzgYcpnXhvSOi0DpYb GuknaYEWRanALBmcvST58xZKjNk/LrJOmWMKK4QnInaFuq18B/TK6jlOgMTMFLnN8JyQ RJzg2BPjm6VP8wQL5U2O+jjO2TsmoLILG1XI3hyYIZEIvQbMqDp6ujHbeDmSbvU5Nqnm Eg== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2ksqrp5hvq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Aug 2018 18:43:52 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w7DIhqX9022544 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Aug 2018 18:43:52 GMT Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w7DIhpXG025325; Mon, 13 Aug 2018 18:43:51 GMT Received: from slnelson-mint18.us.oracle.com (/10.159.144.11) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 13 Aug 2018 11:43:51 -0700 From: Shannon Nelson To: intel-wired-lan@lists.osuosl.org, jeffrey.t.kirsher@intel.com Cc: steffen.klassert@secunet.com, netdev@vger.kernel.org Subject: [PATCH next-queue 0/8] ixgbe/ixgbevf: IPsec offload support for VFs Date: Mon, 13 Aug 2018 11:43:37 -0700 Message-Id: <1534185825-12451-1-git-send-email-shannon.nelson@oracle.com> X-Mailer: git-send-email 2.7.4 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8984 signatures=668707 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808130188 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This set of patches implements IPsec hardware offload for VF devices in Intel's 10Gbe x540 family of Ethernet devices. The IPsec HW offload feature has been in the x540/Niantic family of network devices since their release in 2009, but there was no Linux kernel support for the offload until 2017. After the XFRM code added support for the offload last year, the hw offload was added to the ixgbe PF driver. Since the related x540 VF device uses same setup as the PF for implementing the offload, adding the feature to the ixgbevf seemed like a good idea. In this case, the PF owns the device registers, so the VF simply packages up the request information into a VF<->PF message and the PF does the device configuration. The resulting IPsec throughput is roughly equivalent to what we see in the PF - nearly line-rate, with the expected drop in CPU cycles burned. (I'm not great at performance statistics, I'll let better folks do the actual measurements as they pertain to their own usage) To make use of the capability, first two things are needed: the PF must be told to enable the offload for VFs (it is off by default) and the VF must be trusted. A new ethtool priv-flag for ixgbe is added to control VF offload support. For example: ethtool --set-priv-flags eth0 vf-ipsec on ip link set eth0 vf 1 trust on Once those are set up and the VF device is UP, the user can add SAs the same as for PFs, whether the VF is in the host or has been assigned to a VM. Note that the x540 chip supports a total of 1024 Rx plus 1024 Tx Security Associations (SAs), shared among the PF and VFs that might request them. It is entirely possible for a single VF to soak up all the offload capability, which would likely annoy some people. It seems rather arbitrary to try to set a limit for how many a VF could be allowed, but this is mitigated somewhat by the need for "trust" and "vf-ipsec" to be enabled. I suppose we could come up with a way to make a limit configurable, but there is no existing method for adding that kind configuration so I'll leave that to a future discussion. Currently this doesn't support Tx offload as the hardware encryption engine doesn't seem to engage on the Tx packets. This may be a lingering driver bug, more investigation is needed. Until then, requests for a Tx offload are failed and the userland requester will need to add Tx SAs without the offload attribute. Given that we don't have Tx offload support, the benefit here is less than it could be, but is definitely still noticeable. For example, with informal iperf testing over a 10Gbps link, with full offload in a PF on one side and a VF in a VM on the other side on a CPU with AES instructions: Reference: No IPsec: 9.4 Gbps IPsec offload btwn two PFs: 9.2 Gbps VF as the iperf receiver: IPsec offload on PF, none on VF: 6.8 Gbps IPsec offload on PF and VF: 9.2 Gbps << biggest benefit VF as the iperf sender: IPsec offload on PF, none on VF: 4.8 Gbps IPsec offload on PF and VF: 4.8 Gbps The iperf traffic is primarily uni-directional, and we can see the most benefit when VF is the iperf server and is receiving the test traffic. Watching output from sar also shows a nice decrease in CPU utilization. Shannon Nelson (8): ixgbe: reload ipsec ip table after sa tables ixgbe: prep ipsec constants for later use ixgbe: add VF ipsec management ixgbe: add VF IPsec offload enable flag ixgbe: add VF IPsec offload request message handling ixgbevf: add defines for IPsec offload request ixgbevf: add VF ipsec offload code ixgbevf: enable VF ipsec offload operations drivers/net/ethernet/intel/ixgbe/ixgbe.h | 20 +- drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c | 9 + drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c | 275 ++++++++- drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.h | 13 + drivers/net/ethernet/intel/ixgbe/ixgbe_mbx.h | 5 + drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c | 17 +- drivers/net/ethernet/intel/ixgbevf/Makefile | 1 + drivers/net/ethernet/intel/ixgbevf/defines.h | 10 +- drivers/net/ethernet/intel/ixgbevf/ethtool.c | 2 + drivers/net/ethernet/intel/ixgbevf/ipsec.c | 673 ++++++++++++++++++++++ drivers/net/ethernet/intel/ixgbevf/ipsec.h | 66 +++ drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 33 ++ drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 74 ++- drivers/net/ethernet/intel/ixgbevf/mbx.h | 5 + drivers/net/ethernet/intel/ixgbevf/vf.c | 4 + 15 files changed, 1163 insertions(+), 44 deletions(-) create mode 100644 drivers/net/ethernet/intel/ixgbevf/ipsec.c create mode 100644 drivers/net/ethernet/intel/ixgbevf/ipsec.h