From patchwork Thu Apr 27 08:30:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 755884 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wD9BR1t7Yz9s7f for ; Thu, 27 Apr 2017 18:30:39 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ozlabs-ru.20150623.gappssmtp.com header.i=@ozlabs-ru.20150623.gappssmtp.com header.b="KSwc9V98"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3wD9BR0jXTzDqFb for ; Thu, 27 Apr 2017 18:30:39 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ozlabs-ru.20150623.gappssmtp.com header.i=@ozlabs-ru.20150623.gappssmtp.com header.b="KSwc9V98"; dkim-atps=neutral X-Original-To: slof@lists.ozlabs.org Delivered-To: slof@lists.ozlabs.org Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3wD9BK3L6MzDqFG for ; Thu, 27 Apr 2017 18:30:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ozlabs-ru.20150623.gappssmtp.com header.i=@ozlabs-ru.20150623.gappssmtp.com header.b="KSwc9V98"; dkim-atps=neutral Received: by mail-pf0-x244.google.com with SMTP id c198so7692046pfc.0 for ; Thu, 27 Apr 2017 01:30:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IdWs/t+u0aLVDeRUeDVoy47YBl6JrDt+L+gT8pUezCY=; b=KSwc9V98QbfDme4gDgJK+rglrs9fsaL5+E9AvsxCaDS6k2agYC4ixk7La8CaP5Rslw sfm3h2Qx3mGRaxijNYzMIbqBH9jZ2AwPT6+72cmoUjB1N9v7Cvm8Y3IscfJ8ChwVDM6w DZGtI3reKvTDbLu0WM1XU8nib9bV4pHwvPUy1tWDok9ou0K4/HggIZyQmpr8e1e4Dpsm cUOkT62peZAWKMx6RNnLOOx+Dy+Lmlj/l/N/oC80E/YpM8ayZEBLUw/g/tCD1sDW7Nqa tx8oskclwQun3jGXHBzXlZuu1Z6Ds0iPW8OVBqj2igG7WSWiTndUO/THIlx/2s6Gy2MM AogQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IdWs/t+u0aLVDeRUeDVoy47YBl6JrDt+L+gT8pUezCY=; b=M1rsG/3Eh1txCRcid/3564/q+DexHFGHzsYiVt27+Y4Avul0GZNwNHrt9C2tt/pskY wQWJnO6Kkz1rWvac/Wvg8vU0sYyeoYmJP9l92CjVqovHMje50/cF9n9H6UvfxkTc57tL i+YP8bBzDWogKAWNXHtnHTRDQkh+fRcX1jy+MLeWWyaK+/7MUoUvWk/LbuusvKvwD1Aa lUBQdjdk+ixvsA0T3kJnbhgpzJndwp4LPHaR8YKGmgm121dTr91e5DI/nMjTBmkJpcQO lnmQtbJJ1VZtzJz2Dh1MQ5Al0/yJXOEegaX1ekpEYeUBUZRwIyvA5fGKw96U2QRS9u2w x5Yw== X-Gm-Message-State: AN3rC/5z8EQP0TIS/41zXOeRz1VTANtw9Cs0l1uoosSXCDS/zR1HhTom aF35XO8ySQQQsw== X-Received: by 10.99.147.20 with SMTP id b20mr4503702pge.100.1493281830150; Thu, 27 Apr 2017 01:30:30 -0700 (PDT) Received: from aik.ozlabs.ibm.com ([122.99.82.10]) by smtp.googlemail.com with ESMTPSA id u85sm3088557pfg.64.2017.04.27.01.30.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 27 Apr 2017 01:30:30 -0700 (PDT) Date: Thu, 27 Apr 2017 18:30:22 +1000 From: Alexey Kardashevskiy To: Thomas Huth Message-ID: <20170427183022.54569b63@aik.ozlabs.ibm.com> In-Reply-To: <99870e49-739b-d7e0-1c7e-294989237486@redhat.com> References: <20170424030131.32311-1-aik@ozlabs.ru> <7d1b54d2-fb16-8b69-a3a5-10d84324b83b@redhat.com> <20170427165448.67fa6d60@aik.ozlabs.ibm.com> <99870e49-739b-d7e0-1c7e-294989237486@redhat.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Subject: Re: [SLOF] [PATCH slof] pci: Put non-prefetchable 64bit BARs into 32bit MMIO window X-BeenThere: slof@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Patches for https://github.com/aik/SLOF" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: slof@lists.ozlabs.org Errors-To: slof-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "SLOF" On Thu, 27 Apr 2017 09:50:36 +0200 Thomas Huth wrote: > On 27.04.2017 09:10, Thomas Huth wrote: > > On 27.04.2017 08:54, Alexey Kardashevskiy wrote: > >> On Wed, 26 Apr 2017 15:23:40 +0200 > >> Thomas Huth wrote: > >> > >>> On 24.04.2017 05:01, Alexey Kardashevskiy wrote: > >>>> At the moment 64bit non-prefetchable BARs of devices behind PCI > >>>> p2p bridge go to a 64bit prefetchable windows which is not > >>>> correct and causes linux guests to fail to ioremap() such > >>>> resources. > >>>> > >>>> This moves 64bit non-prefetchable BARs 32bit non-prefetchable > >>>> window. > >>>> > >>>> Note that this does not make distinction between P2P and PHB so > >>>> from now on XHCI BARs will be allocated from 32bit MMIO space. > >>>> However since most 64bit-MMIO-capable devices have prefetchable > >>>> BARs, and XHCI BAR is just 4K (so it is unlikely to cause any > >>>> space problems), this should not affect usual behavior. > >>>> > >>>> Signed-off-by: Alexey Kardashevskiy > >>>> --- > >>>> > >>>> This fixes QEMU's XHCI when it is put on a P2P PCI bridge. > >>>> > >>>> There is a little naming confusion as it may look like the patch > >>>> is changing assignment for all 64bit BAR but it does not as: > >>>> - "mmio" is used for non-prefetchable memory, > >>>> - "mem" is used for prefetchable memory. > >>>> --- > >>>> slof/fs/pci-properties.fs | 6 +----- > >>>> 1 file changed, 1 insertion(+), 5 deletions(-) > >>>> > >>>> diff --git a/slof/fs/pci-properties.fs > >>>> b/slof/fs/pci-properties.fs index e6fd843..8594e5d 100644 > >>>> --- a/slof/fs/pci-properties.fs > >>>> +++ b/slof/fs/pci-properties.fs > >>>> @@ -166,11 +166,7 @@ > >>>> \ Setup a non-prefetchable 64bit BAR and return its size > >>>> : assign-mmio64-bar ( bar-addr -- 8 ) > >>>> dup pci-bar-size-mem64 \ fetch size > >>>> - pci-next-mem64 @ 0 = IF \ Check if we have > >>>> 64-bit memory range > >>>> - pci-next-mmio > >>>> - ELSE > >>>> - pci-next-mem64 \ for board-qemu we will > >>>> use same range > >>>> - THEN > >>>> + pci-next-mmio > >>>> assign-bar-value64 \ and set it all > >>>> ; > >>> > >>> I'm sorry, but this is now causing trouble with the USB keyboard > >>> in *SLOF* instead. If I start QEMU now like this: > >>> > >>> qemu-system-ppc64 -nodefaults -serial mon:stdio \ > >>> -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1 \ > >>> -device nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3 -vga std \ > >>> -device usb-kbd,bus=xhci1.0 -bios boot_rom.bin > >>> > >>> then SLOF complains: "usb-xhci: failed to initialize XHCI > >>> controller" and the keyboard is not working at the SLOF prompt > >>> anymore. > >>> > >>> Any idea why this is happening now? > >> > >> > >> XHCI's usb_hcd_dev::base == 0xc0000000 which is a bus address > >> while it is supposed to be a mapped address, i.e. 800000020000000. > >> > >> It used to work before the patch as pci address is the same as > >> mapped address - 210000000000 - and I am not sure if this is not > >> an accident, could be QEMU's PCI MMIO windows/spacing rework David > >> did some time ago. > >> > >> I am trying to figure out now where usb_hcd_dev::base is set in > >> SLOF, it is tricky :-/ > > > > I think the value comes from usb-setup-hcidev in > > pci-class_0c.fs ... so it's likely that translate-my-address is > > failing now. > > > > ... but thinking about this issue again, I now wonder whether your > > original patch was really the right solution here, since QEMU only > > provides three memory regions to the guest: > > 1) I/O > > 2) 32-bit memory > > 3) 64-bit memory > > There is no distinction between prefetchable and non-prefetchable > > here. So I think the original problem might have been caused by > > something else instead... (not sure about that yet, though) > > Yup, looks like the real culprit is the "ranges" property of the PCI > bridges. This can not deal with 64-bit memory regions yet. I think we > should revert the patch to assign-mmio64-bar and fix > pci-bridge-range-props and friends instead... Try this: Works for me, for both PHB and P2P bridges. > > Thomas > --- Alexey diff --git a/slof/fs/pci-properties.fs b/slof/fs/pci-properties.fs index 8594e5d..66e364d 100644 --- a/slof/fs/pci-properties.fs +++ b/slof/fs/pci-properties.fs @@ -349,7 +349,7 @@ 1 OF gen-io-bar-prop ENDOF \ - IO-BAR 2 OF gen-mem32-bar-prop ENDOF \ - MEM32 3 OF gen-pmem32-bar-prop ENDOF \ - MEM32 prefetchable - 4 OF gen-mem64-bar-prop ENDOF \ - MEM64 + 4 OF gen-mem32-bar-prop ENDOF \ - MEM64 5 OF gen-pmem64-bar-prop ENDOF \ - MEM64 prefetchable ENDCASE \ ESAC ( paddr plen bsize ) ;