From patchwork Tue Aug 18 01:28:56 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roland Dreier X-Patchwork-Id: 31542 Return-Path: X-Original-To: patchwork-incoming@bilbo.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from ozlabs.org (ozlabs.org [203.10.76.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mx.ozlabs.org", Issuer "CA Cert Signing Authority" (verified OK)) by bilbo.ozlabs.org (Postfix) with ESMTPS id 41666B7B66 for ; Tue, 18 Aug 2009 11:29:07 +1000 (EST) Received: by ozlabs.org (Postfix) id 360E0DDDA2; Tue, 18 Aug 2009 11:29:07 +1000 (EST) Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by ozlabs.org (Postfix) with ESMTP id 7C520DDD1B for ; Tue, 18 Aug 2009 11:29:06 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758554AbZHRB24 (ORCPT ); Mon, 17 Aug 2009 21:28:56 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1758533AbZHRB24 (ORCPT ); Mon, 17 Aug 2009 21:28:56 -0400 Received: from sj-iport-2.cisco.com ([171.71.176.71]:55206 "EHLO sj-iport-2.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753889AbZHRB2z (ORCPT ); Mon, 17 Aug 2009 21:28:55 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEAH+giUqrR7MV/2dsb2JhbAC+IYgtj1cFhBmBUg X-IronPort-AV: E=Sophos;i="4.43,399,1246838400"; d="scan'208";a="196302327" Received: from sj-dkim-1.cisco.com ([171.71.179.21]) by sj-iport-2.cisco.com with ESMTP; 18 Aug 2009 01:28:57 +0000 Received: from sj-core-1.cisco.com (sj-core-1.cisco.com [171.71.177.237]) by sj-dkim-1.cisco.com (8.12.11/8.12.11) with ESMTP id n7I1Svxc009302; Mon, 17 Aug 2009 18:28:57 -0700 Received: from xbh-sjc-231.amer.cisco.com (xbh-sjc-231.cisco.com [128.107.191.100]) by sj-core-1.cisco.com (8.13.8/8.14.3) with ESMTP id n7I1SvhW025370; Tue, 18 Aug 2009 01:28:57 GMT Received: from xfe-sjc-212.amer.cisco.com ([171.70.151.187]) by xbh-sjc-231.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959); Mon, 17 Aug 2009 18:28:57 -0700 Received: from roland-conroe ([10.33.42.9]) by xfe-sjc-212.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959); Mon, 17 Aug 2009 18:28:57 -0700 Received: by roland-conroe (Postfix, from userid 33217) id BDF0DE71D7; Mon, 17 Aug 2009 18:28:56 -0700 (PDT) From: Roland Dreier To: Christoph Lameter Cc: netdev@vger.kernel.org, Yevgeny Petrilin Subject: Re: mlx4 2.6.31-rc5: SW2HW_EQ failed. References: X-Message-Flag: Warning: May contain useful information Date: Mon, 17 Aug 2009 18:28:56 -0700 In-Reply-To: (Christoph Lameter's message of "Mon, 17 Aug 2009 18:17:57 -0400 (EDT)") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.91 (gnu/linux) MIME-Version: 1.0 X-OriginalArrivalTime: 18 Aug 2009 01:28:57.0032 (UTC) FILETIME=[415E7480:01CA1FA3] DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; l=5941; t=1250558937; x=1251422937; c=relaxed/simple; s=sjdkim1004; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=rdreier@cisco.com; z=From:=20Roland=20Dreier=20 |Subject:=20Re=3A=20mlx4=202.6.31-rc5=3A=20SW2HW_EQ=20faile d. |Sender:=20; bh=LdeMzNlClynbuwvKZHkZRs1HtZeV962phgm0URvqhrI=; b=BVKs0OMwYTz8Ip9lW8n3FiKp8oqx6X2Gb8LSHWs/LYAnkLzRK2zkF1pG9b 3V2Kvk6BFmX8nSFe+O83K8ijmsZG/PpMz7QYf7B5Qs+7PUS30wMNWli0c83l Ib2/9cvEQD+AFZmSR2pX6E5yrpW+0KRxSifkXPcsPz+gPTHLiqx0k=; Authentication-Results: sj-dkim-1; header.From=rdreier@cisco.com; dkim=pass ( sig from cisco.com/sjdkim1004 verified; ); Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > > [ 10.256371] mlx4_core 0000:04:00.0: SW2HW_EQ failed (-5) > Device FW??? The log you wanted follows at the end of this message. Not sure why there are "???" there... the (-5) error code is an "internal error" status from the device FW on the event queue initialization command. Anyway I think the log shows that the problem is exactly the one fixed in the commit I mentioned -- a423b8a0 ("mlx4_core: Allocate and map sufficient ICM memory for EQ context") from my infiniband.git tree should fix this. The log > [ 7425.199430] mlx4_core 0000:04:00.0: irq 70 for MSI/MSI-X ... > [ 7425.199488] mlx4_core 0000:04:00.0: irq 102 for MSI/MSI-X shows 33 event queues being allocated (num_possible_cpus() + 1) and that will hit the issue fixed in that commit. Assuming this fixes it for you, I guess I should get this into 2.6.31, since it obviously is hitting not-particularly-exotic systems in practice. I do wonder why num_possible_cpus() is 32 on your box (since 16 threads is really the max with nehalem EP). Anyway, here's the patch I mean: commit a423b8a022d523abe834cefe67bfaf42424150a7 Author: Eli Cohen Date: Fri Aug 7 11:13:13 2009 -0700 mlx4_core: Allocate and map sufficient ICM memory for EQ context The current implementation allocates a single host page for EQ context memory, which was OK when we only allocated a few EQs. However, since we now allocate an EQ for each CPU core, this patch removes the hard-coded limit and makes the allocation depend on EQ entry size and the number of required EQs. Signed-off-by: Eli Cohen Signed-off-by: Roland Dreier --- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c index c11a052..dae6387 100644 --- a/drivers/net/mlx4/eq.c +++ b/drivers/net/mlx4/eq.c @@ -529,29 +529,36 @@ int mlx4_map_eq_icm(struct mlx4_dev *dev, u64 icm_virt) { struct mlx4_priv *priv = mlx4_priv(dev); int ret; + int host_pages, icm_pages; + int i; - /* - * We assume that mapping one page is enough for the whole EQ - * context table. This is fine with all current HCAs, because - * we only use 32 EQs and each EQ uses 64 bytes of context - * memory, or 1 KB total. - */ + host_pages = ALIGN(min_t(int, dev->caps.num_eqs, num_possible_cpus() + 1) * + dev->caps.eqc_entry_size, PAGE_SIZE) >> PAGE_SHIFT; + priv->eq_table.order = order_base_2(host_pages); priv->eq_table.icm_virt = icm_virt; - priv->eq_table.icm_page = alloc_page(GFP_HIGHUSER); + priv->eq_table.icm_page = alloc_pages(GFP_HIGHUSER, priv->eq_table.order); if (!priv->eq_table.icm_page) return -ENOMEM; priv->eq_table.icm_dma = pci_map_page(dev->pdev, priv->eq_table.icm_page, 0, - PAGE_SIZE, PCI_DMA_BIDIRECTIONAL); + PAGE_SIZE << priv->eq_table.order, + PCI_DMA_BIDIRECTIONAL); if (pci_dma_mapping_error(dev->pdev, priv->eq_table.icm_dma)) { - __free_page(priv->eq_table.icm_page); + __free_pages(priv->eq_table.icm_page, priv->eq_table.order); return -ENOMEM; } - ret = mlx4_MAP_ICM_page(dev, priv->eq_table.icm_dma, icm_virt); - if (ret) { - pci_unmap_page(dev->pdev, priv->eq_table.icm_dma, PAGE_SIZE, - PCI_DMA_BIDIRECTIONAL); - __free_page(priv->eq_table.icm_page); + icm_pages = (PAGE_SIZE / MLX4_ICM_PAGE_SIZE) << priv->eq_table.order; + for (i = 0; i < icm_pages; ++i) { + ret = mlx4_MAP_ICM_page(dev, priv->eq_table.icm_dma, + icm_virt + i * MLX4_ICM_PAGE_SIZE); + if (ret) { + if (i) + mlx4_UNMAP_ICM(dev, priv->eq_table.icm_virt, i); + pci_unmap_page(dev->pdev, priv->eq_table.icm_dma, PAGE_SIZE, + PCI_DMA_BIDIRECTIONAL); + __free_pages(priv->eq_table.icm_page, priv->eq_table.order); + break; + } } return ret; @@ -560,11 +567,12 @@ int mlx4_map_eq_icm(struct mlx4_dev *dev, u64 icm_virt) void mlx4_unmap_eq_icm(struct mlx4_dev *dev) { struct mlx4_priv *priv = mlx4_priv(dev); + int icm_pages = (PAGE_SIZE / MLX4_ICM_PAGE_SIZE) << priv->eq_table.order; - mlx4_UNMAP_ICM(dev, priv->eq_table.icm_virt, 1); - pci_unmap_page(dev->pdev, priv->eq_table.icm_dma, PAGE_SIZE, - PCI_DMA_BIDIRECTIONAL); - __free_page(priv->eq_table.icm_page); + mlx4_UNMAP_ICM(dev, priv->eq_table.icm_virt, icm_pages); + pci_unmap_page(dev->pdev, priv->eq_table.icm_dma, + PAGE_SIZE << priv->eq_table.order, PCI_DMA_BIDIRECTIONAL); + __free_pages(priv->eq_table.icm_page, priv->eq_table.order); } int mlx4_alloc_eq_table(struct mlx4_dev *dev) diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 5c1afe0..474d1f3 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -207,6 +207,7 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) dev->caps.max_cqes = dev_cap->max_cq_sz - 1; dev->caps.reserved_cqs = dev_cap->reserved_cqs; dev->caps.reserved_eqs = dev_cap->reserved_eqs; + dev->caps.eqc_entry_size = dev_cap->eqc_entry_sz; dev->caps.mtts_per_seg = 1 << log_mtts_per_seg; dev->caps.reserved_mtts = DIV_ROUND_UP(dev_cap->reserved_mtts, dev->caps.mtts_per_seg); diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index 5bd79c2..34bcc11 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -210,6 +210,7 @@ struct mlx4_eq_table { dma_addr_t icm_dma; struct mlx4_icm_table cmpt_table; int have_irq; + int order; u8 inta_pin; }; diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index ce7cc6c..8923c9b 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -206,6 +206,7 @@ struct mlx4_caps { int max_cqes; int reserved_cqs; int num_eqs; + int eqc_entry_size; int reserved_eqs; int num_comp_vectors; int num_mpts;