From patchwork Wed Mar 14 10:01:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ursula Braun X-Patchwork-Id: 885655 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 401S0v3pDTz9sTp for ; Wed, 14 Mar 2018 21:01:19 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751673AbeCNKBR (ORCPT ); Wed, 14 Mar 2018 06:01:17 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:42018 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751426AbeCNKBL (ORCPT ); Wed, 14 Mar 2018 06:01:11 -0400 Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2E9xCP6064253 for ; Wed, 14 Mar 2018 06:01:11 -0400 Received: from e06smtp10.uk.ibm.com (e06smtp10.uk.ibm.com [195.75.94.106]) by mx0b-001b2d01.pphosted.com with ESMTP id 2gpxcrf7wg-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Wed, 14 Mar 2018 06:01:10 -0400 Received: from localhost by e06smtp10.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 14 Mar 2018 10:01:09 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp10.uk.ibm.com (192.168.101.140) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 14 Mar 2018 10:01:06 -0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w2EA16nm57671866; Wed, 14 Mar 2018 10:01:06 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 822694204C; Wed, 14 Mar 2018 09:53:21 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1ED0A42041; Wed, 14 Mar 2018 09:53:21 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Wed, 14 Mar 2018 09:53:21 +0000 (GMT) From: Ursula Braun To: davem@davemloft.net Cc: netdev@vger.kernel.org, linux-s390@vger.kernel.org, linux-rdma@vger.kernel.org, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, raspl@linux.vnet.ibm.com, ubraun@linux.vnet.ibm.com Subject: [PATCH net-next 1/3] net/smc: pay attention to MAX_ORDER for CQ entries Date: Wed, 14 Mar 2018 11:01:00 +0100 X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180314100102.40474-1-ubraun@linux.vnet.ibm.com> References: <20180314100102.40474-1-ubraun@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18031410-0040-0000-0000-000004204BFD X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18031410-0041-0000-0000-000026234E91 Message-Id: <20180314100102.40474-2-ubraun@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-03-14_05:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1803140117 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org smc allocates a certain number of CQ entries for used RoCE devices. For mlx5 devices the chosen constant number results in a large allocation causing this warning: [13355.124656] WARNING: CPU: 3 PID: 16535 at mm/page_alloc.c:3883 __alloc_pages_nodemask+0x2be/0x10c0 [13355.124657] Modules linked in: smc_diag(O) smc(O) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter mlx5_ib ib_core sunrpc mlx5_core s390_trng rng_core ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common ptp pps_core eadm_sch dm_multipath dm_mod vhost_net tun vhost tap sch_fq_codel kvm ip_tables x_tables autofs4 [last unloaded: smc] [13355.124672] CPU: 3 PID: 16535 Comm: kworker/3:0 Tainted: G O 4.14.0uschi #1 [13355.124673] Hardware name: IBM 3906 M04 704 (LPAR) [13355.124675] Workqueue: events smc_listen_work [smc] [13355.124677] task: 00000000e2f22100 task.stack: 0000000084720000 [13355.124678] Krnl PSW : 0704c00180000000 000000000029da76 (__alloc_pages_nodemask+0x2be/0x10c0) [13355.124681] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3 [13355.124682] Krnl GPRS: 0000000000000000 00550e00014080c0 0000000000000000 0000000000000001 [13355.124684] 000000000029d8b6 00000000f3bfd710 0000000000000000 00000000014080c0 [13355.124685] 0000000000000009 00000000ec277a00 0000000000200000 0000000000000000 [13355.124686] 0000000000000000 00000000000001ff 000000000029d8b6 0000000084723720 [13355.124708] Krnl Code: 000000000029da6a: a7110200 tmll %r1,512 000000000029da6e: a774ff29 brc 7,29d8c0 #000000000029da72: a7f40001 brc 15,29da74 >000000000029da76: a7f4ff25 brc 15,29d8c0 000000000029da7a: a7380000 lhi %r3,0 000000000029da7e: a7f4fef1 brc 15,29d860 000000000029da82: 5820f0c4 l %r2,196(%r15) 000000000029da86: a53e0048 llilh %r3,72 [13355.124720] Call Trace: [13355.124722] ([<000000000029d8b6>] __alloc_pages_nodemask+0xfe/0x10c0) [13355.124724] [<000000000013bd1e>] s390_dma_alloc+0x6e/0x148 [13355.124733] [<000003ff802eeba6>] mlx5_dma_zalloc_coherent_node+0x8e/0xe0 [mlx5_core] [13355.124740] [<000003ff802eee18>] mlx5_buf_alloc_node+0x70/0x108 [mlx5_core] [13355.124744] [<000003ff804eb410>] mlx5_ib_create_cq+0x558/0x898 [mlx5_ib] [13355.124749] [<000003ff80407d40>] ib_create_cq+0x48/0x88 [ib_core] [13355.124751] [<000003ff80109fba>] smc_ib_setup_per_ibdev+0x52/0x118 [smc] [13355.124753] [<000003ff8010bcb6>] smc_conn_create+0x65e/0x728 [smc] [13355.124755] [<000003ff801081a2>] smc_listen_work+0x2d2/0x540 [smc] [13355.124756] [<0000000000162c66>] process_one_work+0x1be/0x440 [13355.124758] [<0000000000162f40>] worker_thread+0x58/0x458 [13355.124759] [<0000000000169e7e>] kthread+0x14e/0x168 [13355.124760] [<00000000009ce8be>] kernel_thread_starter+0x6/0xc [13355.124762] [<00000000009ce8b8>] kernel_thread_starter+0x0/0xc [13355.124762] Last Breaking-Event-Address: [13355.124764] [<000000000029da72>] __alloc_pages_nodemask+0x2ba/0x10c0 [13355.124764] ---[ end trace 34be38b581c0b585 ]--- This patch reduces the smc constant for the maximum number of allocated completion queue entries SMC_MAX_CQE by 2 to avoid high round up values in the mlx5 code, and reduces the number of allocated completion queue entries even more, if the final allocation for an mlx5 device hits the MAX_ORDER limit. Reported-by: Ihnken Menssen Signed-off-by: Ursula Braun --- net/smc/smc_ib.c | 10 +++++++++- net/smc/smc_wr.h | 1 - 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/net/smc/smc_ib.c b/net/smc/smc_ib.c index 2a8957bd6d38..26df554f7588 100644 --- a/net/smc/smc_ib.c +++ b/net/smc/smc_ib.c @@ -23,6 +23,8 @@ #include "smc_wr.h" #include "smc.h" +#define SMC_MAX_CQE 32766 /* max. # of completion queue elements */ + #define SMC_QP_MIN_RNR_TIMER 5 #define SMC_QP_TIMEOUT 15 /* 4096 * 2 ** timeout usec */ #define SMC_QP_RETRY_CNT 7 /* 7: infinite */ @@ -438,9 +440,15 @@ int smc_ib_remember_port_attr(struct smc_ib_device *smcibdev, u8 ibport) long smc_ib_setup_per_ibdev(struct smc_ib_device *smcibdev) { struct ib_cq_init_attr cqattr = { - .cqe = SMC_WR_MAX_CQE, .comp_vector = 0 }; + .cqe = SMC_MAX_CQE, .comp_vector = 0 }; + int cqe_size_order, smc_order; long rc; + /* the calculated number of cq entries fits to mlx5 cq allocation */ + cqe_size_order = cache_line_size() == 128 ? 7 : 6; + smc_order = MAX_ORDER - cqe_size_order - 1; + if (SMC_MAX_CQE + 2 > (0x00000001 << smc_order) * PAGE_SIZE) + cqattr.cqe = (0x00000001 << smc_order) * PAGE_SIZE - 2; smcibdev->roce_cq_send = ib_create_cq(smcibdev->ibdev, smc_wr_tx_cq_handler, NULL, smcibdev, &cqattr); diff --git a/net/smc/smc_wr.h b/net/smc/smc_wr.h index ef0c3494c9cb..210bec3c3ebe 100644 --- a/net/smc/smc_wr.h +++ b/net/smc/smc_wr.h @@ -19,7 +19,6 @@ #include "smc.h" #include "smc_core.h" -#define SMC_WR_MAX_CQE 32768 /* max. # of completion queue elements */ #define SMC_WR_BUF_CNT 16 /* # of ctrl buffers per link */ #define SMC_WR_TX_WAIT_FREE_SLOT_TIME (10 * HZ)