From patchwork Thu Sep 13 03:37:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Drake X-Patchwork-Id: 969243 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=endlessm.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=endlessm-com.20150623.gappssmtp.com header.i=@endlessm-com.20150623.gappssmtp.com header.b="iFgQ5mXr"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 429kr505wJz9sB4 for ; Thu, 13 Sep 2018 13:37:57 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726680AbeIMIp2 (ORCPT ); Thu, 13 Sep 2018 04:45:28 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:33328 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726500AbeIMIp2 (ORCPT ); Thu, 13 Sep 2018 04:45:28 -0400 Received: by mail-pf1-f194.google.com with SMTP id d4-v6so2024508pfn.0 for ; Wed, 12 Sep 2018 20:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessm-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=DMGmlWmI+qr1DGG2OGhPDIFdibeVcSrncznOavL2acU=; b=iFgQ5mXrcu2MdlZpAlLwBn85lG3vwikTK3+x6eKkf8l9P6nOIuO+06GxngH7iDAtdb 8wocjORdaf4wKLAbeUuEYkP+FIk3YBAINfwUjV+pANicjGtP35IeE0ywu+686aju+j23 LzLa8RW6xzvyiOo9Z4gQsh+lqVOyjlhzbtSugUwJjjeCGHDVY5x45nTf9I5YJdcCIiUt m1+J6Qb6gLvBl/AfsnosW/FbL2sFTDgu4IL0M8gUD7PRKgEhOH2GAEW6k5M3vtrri5Gk 4+rCqM3veCFfWpgUx1Ss+DhJNxRiPkJ4yQn3L4GqRUzNgwf9rkzcc3aKT0FxfC+CUzhz 2ltg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=DMGmlWmI+qr1DGG2OGhPDIFdibeVcSrncznOavL2acU=; b=p9U+RfsskdC/qDemgSH9rO5s7rcJwjGRiIFd9jOxOJ/Mau0vXPbIHY4nMP1gbwh6so LJK+suRv0j7BIi3EdFg/J/8VcVPNo+eJE+7aCjRa7ZCGqeNV7ovIavj7NaOHkqOd9gpr KnYYAP4fvt1zFeZphpf6fiNStzoVNpiJg3Iy7UyHlyiIfL1C5bAw1qZbb2fnkOPvXHG0 ytaNw2ixk9yuxHF4hNusGEInAyCDsww2ngYhUdl4UXQ7zu/8l+LyYa3DjvMFoN2/Whm/ TaxOu2rdUCx/HwXTfCeX9s6SYJJJ5abLcnq7bjkG3XmkPoztE1SQGTG2Ww6UVPQ07paR IVdg== X-Gm-Message-State: APzg51DQAPT7kCenK3YVLEIX0TF5Cyr+YbPqvZp9mw5KLN/eQ6yduOSD WW+lEUldqtnAi4ThWvpt9lp+sw== X-Google-Smtp-Source: ANB0VdaVETA03qKnsE3aB2EEQ6blUpKwmpthLE9z2ROKa0wZwGcmKNJ5rLEWsxz9v9Iaw/1sgQ7UDA== X-Received: by 2002:a62:46c8:: with SMTP id o69-v6mr5399422pfi.21.1536809873384; Wed, 12 Sep 2018 20:37:53 -0700 (PDT) Received: from limbo.local (125-227-158-176.HINET-IP.hinet.net. [125.227.158.176]) by smtp.gmail.com with ESMTPSA id z17-v6sm4311084pfl.146.2018.09.12.20.37.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Sep 2018 20:37:52 -0700 (PDT) From: Daniel Drake To: bhelgaas@google.com Cc: linux-pci@vger.kernel.org, linux@endlessm.com, nouveau@lists.freedesktop.org, linux-pm@vger.kernel.org, peter@lekensteyn.nl, kherbst@redhat.com, andy.shevchenko@linux.intel.com, rafael.j.wysocki@intel.com, keith.busch@intel.com, jonathan.derrick@intel.com, kugel@rockbox.org, davem@davemloft.net, hkallweit1@gmail.com, netdev@vger.kernel.org, nic_swsd@realtek.com, rchang@marvell.com Subject: [PATCH v3] PCI: Reprogram bridge prefetch registers on resume Date: Thu, 13 Sep 2018 11:37:45 +0800 Message-Id: <20180913033745.11178-1-drake@endlessm.com> X-Mailer: git-send-email 2.17.1 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 38+ Intel-based Asus products, the nvidia GPU becomes unusable after S3 suspend/resume. The affected products include multiple generations of nvidia GPUs and Intel SoCs. After resume, nouveau logs many errors such as: fifo: fault 00 [READ] at 0000005555555000 engine 00 [GR] client 04 [HUB/FE] reason 4a [] on channel -1 [007fa91000 unknown] DRM: failed to idle channel 0 [DRM] Similarly, the nvidia proprietary driver also fails after resume (black screen, 100% CPU usage in Xorg process). We shipped a sample to Nvidia for diagnosis, and their response indicated that it's a problem with the parent PCI bridge (on the Intel SoC), not the GPU. Runtime suspend/resume works fine, only S3 suspend is affected. We found a workaround: on resume, rewrite the Intel PCI bridge 'Prefetchable Base Upper 32 Bits' register (PCI_PREF_BASE_UPPER32). In the cases that I checked, this register has value 0 and we just have to rewrite that value. Linux already saves and restores PCI config space during suspend/resume, but this register was being skipped because upon resume, it already has value 0 (the correct, pre-suspend value). Intel appear to have previously acknowledged this behaviour and the requirement to rewrite this register. https://bugzilla.kernel.org/show_bug.cgi?id=116851#c23 Based on that, rewrite the prefetch register values even when that appears unnecessary. We have confirmed this solution on all the affected models we have in-hands (X542UQ, UX533FD, X530UN, V272UN). Additionally, this solves an issue where r8169 MSI-X interrupts were broken after S3 suspend/resume on Asus X441UAR. This issue was recently worked around in commit 7bb05b85bc2d ("r8169: don't use MSI-X on RTL8106e"). It also fixes the same issue on RTL6186evl/8111evl on an Aimfor-tech laptop that we had not yet patched. I suspect it will also fix the issue that was worked around in commit 7c53a722459c ("r8169: don't use MSI-X on RTL8168g"). Thomas Martitz reports that this change also solves an issue where the AMD Radeon Polaris 10 GPU on the HP Zbook 14u G5 is unresponsive after S3 suspend/resume. Link: https://bugzilla.kernel.org/show_bug.cgi?id=201069 Signed-off-by: Daniel Drake Reviewed-by: Rafael J. Wysocki Reviewed-By: Peter Wu --- drivers/pci/pci.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 29ff9619b5fa..5d58220b6997 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1289,12 +1289,12 @@ int pci_save_state(struct pci_dev *dev) EXPORT_SYMBOL(pci_save_state); static void pci_restore_config_dword(struct pci_dev *pdev, int offset, - u32 saved_val, int retry) + u32 saved_val, int retry, bool force) { u32 val; pci_read_config_dword(pdev, offset, &val); - if (val == saved_val) + if (!force && val == saved_val) return; for (;;) { @@ -1313,25 +1313,34 @@ static void pci_restore_config_dword(struct pci_dev *pdev, int offset, } static void pci_restore_config_space_range(struct pci_dev *pdev, - int start, int end, int retry) + int start, int end, int retry, + bool force) { int index; for (index = end; index >= start; index--) pci_restore_config_dword(pdev, 4 * index, pdev->saved_config_space[index], - retry); + retry, force); } static void pci_restore_config_space(struct pci_dev *pdev) { if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL) { - pci_restore_config_space_range(pdev, 10, 15, 0); + pci_restore_config_space_range(pdev, 10, 15, 0, false); /* Restore BARs before the command register. */ - pci_restore_config_space_range(pdev, 4, 9, 10); - pci_restore_config_space_range(pdev, 0, 3, 0); + pci_restore_config_space_range(pdev, 4, 9, 10, false); + pci_restore_config_space_range(pdev, 0, 3, 0, false); + } else if (pdev->hdr_type == PCI_HEADER_TYPE_BRIDGE) { + pci_restore_config_space_range(pdev, 12, 15, 0, false); + /* Force rewriting of prefetch registers to avoid + * S3 resume issues on Intel PCI bridges that occur when + * these registers are not explicitly written. + */ + pci_restore_config_space_range(pdev, 9, 11, 0, true); + pci_restore_config_space_range(pdev, 0, 8, 0, false); } else { - pci_restore_config_space_range(pdev, 0, 15, 0); + pci_restore_config_space_range(pdev, 0, 15, 0, false); } }