Message ID | 823b58f1-f17e-6645-11be-406ad1494e06@ozlabs.ru |
---|---|
State | New |
Headers | show |
Series | virtio capabilities | expand |
On Fri, Dec 13, 2019 at 05:05:05PM +1100, Alexey Kardashevskiy wrote: > Hi! > > I am having an issue with capabilities (hopefully the chunk formatting > won't break). > > The problem is that when virtio_pci_find_capability() reads > pci_find_capability(dev, PCI_CAP_ID_VNDR), 0 is returned; if repeated, > it returns a valid number (0x84). Timing seems to matter. pci_cfg_read > trace shows that that first time read does not reach QEMU but others do > reach QEMU and return what is expected. > > How to debug this, any quick ideas? > The config space is not a MMIO BAR > or KVM memory slot or anything like this, right? :) Thanks, Depends on the platform. E.g. on x86, when using cf8/cfc pair, if guest doesn't have a lock around programming the pair of registers, then one access can conflict with another one. When using express it's MMIO so shouldn't be a problem. > > [ 3.489492] ___K___ (0) virtio_pci_modern_probe 642 > [ 3.489697] ___K___ (0) virtio_pci_find_capability 492: FIND a cap > [ 3.490070] ___K___ (0) virtio_pci_find_capability 494: cap is at 0 > [ 3.490335] ___K___ (0) virtio_pci_find_capability 492: FIND a cap > 10909@1576216763.643271:pci_cfg_read virtio-net-pci 00:0 @0x6 -> 0x10 > 10909@1576216763.643431:pci_cfg_read virtio-net-pci 00:0 @0x34 -> 0x98 > 10909@1576216763.643591:pci_cfg_read virtio-net-pci 00:0 @0x98 -> 0x8411 > 10909@1576216763.643747:pci_cfg_read virtio-net-pci 00:0 @0x84 -> 0x7009 > [ 3.491264] ___K___ (0) virtio_pci_find_capability 494: cap is at 132 > 10909@1576216763.644140:pci_cfg_read virtio-net-pci 00:0 @0x87 -> 0x5 > 10909@1576216763.644287:pci_cfg_read virtio-net-pci 00:0 @0x88 -> 0x0 > [ 3.491803] ___K___ (0) virtio_pci_find_capability 506: 5 0 > 10909@1576216763.644632:pci_cfg_read virtio-net-pci 00:0 @0x85 -> 0x70 > 10909@1576216763.644786:pci_cfg_read virtio-net-pci 00:0 @0x70 -> 0x6009 > 10909@1576216763.644942:pci_cfg_read virtio-net-pci 00:0 @0x73 -> 0x2 > 10909@1576216763.645092:pci_cfg_read virtio-net-pci 00:0 @0x74 -> 0x4 > [ 3.492607] ___K___ (0) virtio_pci_find_capability 506: 2 4 > > > > > > diff --git a/drivers/virtio/virtio_pci_modern.c > b/drivers/virtio/virtio_pci_modern.c > index 7abcc50838b8..85b2a7ce96e9 100644 > --- a/drivers/virtio/virtio_pci_modern.c > +++ b/drivers/virtio/virtio_pci_modern.c > @@ -486,9 +486,14 @@ static const struct virtio_config_ops > virtio_pci_config_ops = { > static inline int virtio_pci_find_capability(struct pci_dev *dev, u8 > cfg_type, > u32 ioresource_types, int > *bars) > { > - int pos; > + int pos = 0;// = pci_find_capability(dev, PCI_CAP_ID_VNDR); > > - for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); > + while (!pos) { > + pr_err("___K___ (%u) %s %u: FIND a cap\n", > smp_processor_id(), __func__, __LINE__); > + pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); > + pr_err("___K___ (%u) %s %u: cap is at %d\n", > smp_processor_id(), __func__, __LINE__, pos); > + } > + for (; > pos > 0; > pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) { > u8 type, bar; > > > -- > Alexey
On 13/12/2019 18:24, Michael S. Tsirkin wrote: > On Fri, Dec 13, 2019 at 05:05:05PM +1100, Alexey Kardashevskiy wrote: >> Hi! >> >> I am having an issue with capabilities (hopefully the chunk formatting >> won't break). >> >> The problem is that when virtio_pci_find_capability() reads >> pci_find_capability(dev, PCI_CAP_ID_VNDR), 0 is returned; if repeated, >> it returns a valid number (0x84). Timing seems to matter. pci_cfg_read >> trace shows that that first time read does not reach QEMU but others do >> reach QEMU and return what is expected. >> >> How to debug this, any quick ideas? >> The config space is not a MMIO BAR >> or KVM memory slot or anything like this, right? :) Thanks, > > Depends on the platform. > > E.g. on x86, when using cf8/cfc pair, if guest doesn't Is there an easy way to tell if it is this "cf8/cfc" case? I have these bars, is any of them related to cf8/cfc? Thanks, root@le-dbg:~# (qemu) info mtree -f FlatView #0 AS "memory", root: system AS "cpu-memory-0", root: system Root memory region: system 0000000000000000-00000000ffffffff (prio 0, ram): ppc_spapr.ram kvm 0000200080000000-000020008000002f (prio 0, i/o): msix-table 0000200080000800-0000200080000807 (prio 0, i/o): msix-pba 0000210000000000-0000210000000fff (prio 0, i/o): virtio-pci-common 0000210000001000-0000210000001fff (prio 0, i/o): virtio-pci-isr 0000210000002000-0000210000002fff (prio 0, i/o): virtio-pci-device 0000210000003000-0000210000003fff (prio 0, i/o): virtio-pci-notify > have a lock around programming the pair of registers, > then one access can conflict with another one. > > When using express it's MMIO so shouldn't be a problem. > >> >> [ 3.489492] ___K___ (0) virtio_pci_modern_probe 642 >> [ 3.489697] ___K___ (0) virtio_pci_find_capability 492: FIND a cap >> [ 3.490070] ___K___ (0) virtio_pci_find_capability 494: cap is at 0 >> [ 3.490335] ___K___ (0) virtio_pci_find_capability 492: FIND a cap >> 10909@1576216763.643271:pci_cfg_read virtio-net-pci 00:0 @0x6 -> 0x10 >> 10909@1576216763.643431:pci_cfg_read virtio-net-pci 00:0 @0x34 -> 0x98 >> 10909@1576216763.643591:pci_cfg_read virtio-net-pci 00:0 @0x98 -> 0x8411 >> 10909@1576216763.643747:pci_cfg_read virtio-net-pci 00:0 @0x84 -> 0x7009 >> [ 3.491264] ___K___ (0) virtio_pci_find_capability 494: cap is at 132 >> 10909@1576216763.644140:pci_cfg_read virtio-net-pci 00:0 @0x87 -> 0x5 >> 10909@1576216763.644287:pci_cfg_read virtio-net-pci 00:0 @0x88 -> 0x0 >> [ 3.491803] ___K___ (0) virtio_pci_find_capability 506: 5 0 >> 10909@1576216763.644632:pci_cfg_read virtio-net-pci 00:0 @0x85 -> 0x70 >> 10909@1576216763.644786:pci_cfg_read virtio-net-pci 00:0 @0x70 -> 0x6009 >> 10909@1576216763.644942:pci_cfg_read virtio-net-pci 00:0 @0x73 -> 0x2 >> 10909@1576216763.645092:pci_cfg_read virtio-net-pci 00:0 @0x74 -> 0x4 >> [ 3.492607] ___K___ (0) virtio_pci_find_capability 506: 2 4 >> >> >> >> >> >> diff --git a/drivers/virtio/virtio_pci_modern.c >> b/drivers/virtio/virtio_pci_modern.c >> index 7abcc50838b8..85b2a7ce96e9 100644 >> --- a/drivers/virtio/virtio_pci_modern.c >> +++ b/drivers/virtio/virtio_pci_modern.c >> @@ -486,9 +486,14 @@ static const struct virtio_config_ops >> virtio_pci_config_ops = { >> static inline int virtio_pci_find_capability(struct pci_dev *dev, u8 >> cfg_type, >> u32 ioresource_types, int >> *bars) >> { >> - int pos; >> + int pos = 0;// = pci_find_capability(dev, PCI_CAP_ID_VNDR); >> >> - for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); >> + while (!pos) { >> + pr_err("___K___ (%u) %s %u: FIND a cap\n", >> smp_processor_id(), __func__, __LINE__); >> + pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); >> + pr_err("___K___ (%u) %s %u: cap is at %d\n", >> smp_processor_id(), __func__, __LINE__, pos); >> + } >> + for (; >> pos > 0; >> pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) { >> u8 type, bar; >> >> >> -- >> Alexey >
On Fri, Dec 13, 2019 at 07:29:40PM +1100, Alexey Kardashevskiy wrote: > > > On 13/12/2019 18:24, Michael S. Tsirkin wrote: > > On Fri, Dec 13, 2019 at 05:05:05PM +1100, Alexey Kardashevskiy wrote: > >> Hi! > >> > >> I am having an issue with capabilities (hopefully the chunk formatting > >> won't break). > >> > >> The problem is that when virtio_pci_find_capability() reads > >> pci_find_capability(dev, PCI_CAP_ID_VNDR), 0 is returned; if repeated, > >> it returns a valid number (0x84). Timing seems to matter. pci_cfg_read > >> trace shows that that first time read does not reach QEMU but others do > >> reach QEMU and return what is expected. > >> > >> How to debug this, any quick ideas? > >> The config space is not a MMIO BAR > >> or KVM memory slot or anything like this, right? :) Thanks, > > > > Depends on the platform. > > > > E.g. on x86, when using cf8/cfc pair, if guest doesn't > > > Is there an easy way to tell if it is this "cf8/cfc" case? > > I have these bars, is any of them related to cf8/cfc? Thanks, > > root@le-dbg:~# (qemu) info mtree -f > FlatView #0 > AS "memory", root: system > AS "cpu-memory-0", root: system > Root memory region: system > 0000000000000000-00000000ffffffff (prio 0, ram): ppc_spapr.ram kvm > 0000200080000000-000020008000002f (prio 0, i/o): msix-table > 0000200080000800-0000200080000807 (prio 0, i/o): msix-pba > 0000210000000000-0000210000000fff (prio 0, i/o): virtio-pci-common > 0000210000001000-0000210000001fff (prio 0, i/o): virtio-pci-isr > 0000210000002000-0000210000002fff (prio 0, i/o): virtio-pci-device > 0000210000003000-0000210000003fff (prio 0, i/o): virtio-pci-notify > No, you want stuff in hw/ppc/spapr_pci.c > > > > have a lock around programming the pair of registers, > > then one access can conflict with another one. > > > > When using express it's MMIO so shouldn't be a problem. > > > >> > >> [ 3.489492] ___K___ (0) virtio_pci_modern_probe 642 > >> [ 3.489697] ___K___ (0) virtio_pci_find_capability 492: FIND a cap > >> [ 3.490070] ___K___ (0) virtio_pci_find_capability 494: cap is at 0 > >> [ 3.490335] ___K___ (0) virtio_pci_find_capability 492: FIND a cap > >> 10909@1576216763.643271:pci_cfg_read virtio-net-pci 00:0 @0x6 -> 0x10 > >> 10909@1576216763.643431:pci_cfg_read virtio-net-pci 00:0 @0x34 -> 0x98 > >> 10909@1576216763.643591:pci_cfg_read virtio-net-pci 00:0 @0x98 -> 0x8411 > >> 10909@1576216763.643747:pci_cfg_read virtio-net-pci 00:0 @0x84 -> 0x7009 > >> [ 3.491264] ___K___ (0) virtio_pci_find_capability 494: cap is at 132 > >> 10909@1576216763.644140:pci_cfg_read virtio-net-pci 00:0 @0x87 -> 0x5 > >> 10909@1576216763.644287:pci_cfg_read virtio-net-pci 00:0 @0x88 -> 0x0 > >> [ 3.491803] ___K___ (0) virtio_pci_find_capability 506: 5 0 > >> 10909@1576216763.644632:pci_cfg_read virtio-net-pci 00:0 @0x85 -> 0x70 > >> 10909@1576216763.644786:pci_cfg_read virtio-net-pci 00:0 @0x70 -> 0x6009 > >> 10909@1576216763.644942:pci_cfg_read virtio-net-pci 00:0 @0x73 -> 0x2 > >> 10909@1576216763.645092:pci_cfg_read virtio-net-pci 00:0 @0x74 -> 0x4 > >> [ 3.492607] ___K___ (0) virtio_pci_find_capability 506: 2 4 > >> > >> > >> > >> > >> > >> diff --git a/drivers/virtio/virtio_pci_modern.c > >> b/drivers/virtio/virtio_pci_modern.c > >> index 7abcc50838b8..85b2a7ce96e9 100644 > >> --- a/drivers/virtio/virtio_pci_modern.c > >> +++ b/drivers/virtio/virtio_pci_modern.c > >> @@ -486,9 +486,14 @@ static const struct virtio_config_ops > >> virtio_pci_config_ops = { > >> static inline int virtio_pci_find_capability(struct pci_dev *dev, u8 > >> cfg_type, > >> u32 ioresource_types, int > >> *bars) > >> { > >> - int pos; > >> + int pos = 0;// = pci_find_capability(dev, PCI_CAP_ID_VNDR); > >> > >> - for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); > >> + while (!pos) { > >> + pr_err("___K___ (%u) %s %u: FIND a cap\n", > >> smp_processor_id(), __func__, __LINE__); > >> + pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); > >> + pr_err("___K___ (%u) %s %u: cap is at %d\n", > >> smp_processor_id(), __func__, __LINE__, pos); > >> + } > >> + for (; > >> pos > 0; > >> pos = pci_find_next_capability(dev, pos, PCI_CAP_ID_VNDR)) { > >> u8 type, bar; > >> > >> > >> -- > >> Alexey > > > > -- > Alexey
On 13/12/2019 19:36, Michael S. Tsirkin wrote: > On Fri, Dec 13, 2019 at 07:29:40PM +1100, Alexey Kardashevskiy wrote: >> >> >> On 13/12/2019 18:24, Michael S. Tsirkin wrote: >>> On Fri, Dec 13, 2019 at 05:05:05PM +1100, Alexey Kardashevskiy wrote: >>>> Hi! >>>> >>>> I am having an issue with capabilities (hopefully the chunk formatting >>>> won't break). >>>> >>>> The problem is that when virtio_pci_find_capability() reads >>>> pci_find_capability(dev, PCI_CAP_ID_VNDR), 0 is returned; if repeated, >>>> it returns a valid number (0x84). Timing seems to matter. pci_cfg_read >>>> trace shows that that first time read does not reach QEMU but others do >>>> reach QEMU and return what is expected. >>>> >>>> How to debug this, any quick ideas? >>>> The config space is not a MMIO BAR >>>> or KVM memory slot or anything like this, right? :) Thanks, >>> >>> Depends on the platform. >>> >>> E.g. on x86, when using cf8/cfc pair, if guest doesn't >> >> >> Is there an easy way to tell if it is this "cf8/cfc" case? >> >> I have these bars, is any of them related to cf8/cfc? Thanks, >> >> root@le-dbg:~# (qemu) info mtree -f >> FlatView #0 >> AS "memory", root: system >> AS "cpu-memory-0", root: system >> Root memory region: system >> 0000000000000000-00000000ffffffff (prio 0, ram): ppc_spapr.ram kvm >> 0000200080000000-000020008000002f (prio 0, i/o): msix-table >> 0000200080000800-0000200080000807 (prio 0, i/o): msix-pba >> 0000210000000000-0000210000000fff (prio 0, i/o): virtio-pci-common >> 0000210000001000-0000210000001fff (prio 0, i/o): virtio-pci-isr >> 0000210000002000-0000210000002fff (prio 0, i/o): virtio-pci-device >> 0000210000003000-0000210000003fff (prio 0, i/o): virtio-pci-notify >> > > > No, you want stuff in hw/ppc/spapr_pci.c The problem was with our firmware, fixing that now. Out of curiosity. I do not see cf8/cfc on x86 either, or I just do not recognize those, what is this cf8/cfc? Thanks, FlatView #2 AS "memory", root: system AS "cpu-memory-0", root: system AS "piix3-ide", root: bus master container AS "virtio-net-pci", root: bus master container Root memory region: system 0000000000000000-00000000000bffff (prio 0, ram): pc.ram kvm 00000000000c0000-00000000000c0fff (prio 0, rom): pc.ram @00000000000c0000 kvm 00000000000c1000-00000000000c3fff (prio 0, ram): pc.ram @00000000000c1000 kvm 00000000000c4000-00000000000e7fff (prio 0, rom): pc.ram @00000000000c4000 kvm 00000000000e8000-00000000000effff (prio 0, ram): pc.ram @00000000000e8000 kvm 00000000000f0000-00000000000fffff (prio 0, rom): pc.ram @00000000000f0000 kvm 0000000000100000-000000007fffffff (prio 0, ram): pc.ram @0000000000100000 kvm 00000000febc0000-00000000febc002f (prio 0, i/o): msix-table 00000000febc0800-00000000febc0807 (prio 0, i/o): msix-pba 00000000febfc000-00000000febfcfff (prio 0, i/o): virtio-pci-common 00000000febfd000-00000000febfdfff (prio 0, i/o): virtio-pci-isr 00000000febfe000-00000000febfefff (prio 0, i/o): virtio-pci-device 00000000febff000-00000000febfffff (prio 0, i/o): virtio-pci-notify 00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios kvm
On Wed, Dec 18, 2019 at 04:19:57PM +1100, Alexey Kardashevskiy wrote: > > > On 13/12/2019 19:36, Michael S. Tsirkin wrote: > > On Fri, Dec 13, 2019 at 07:29:40PM +1100, Alexey Kardashevskiy wrote: > >> > >> > >> On 13/12/2019 18:24, Michael S. Tsirkin wrote: > >>> On Fri, Dec 13, 2019 at 05:05:05PM +1100, Alexey Kardashevskiy wrote: > >>>> Hi! > >>>> > >>>> I am having an issue with capabilities (hopefully the chunk formatting > >>>> won't break). > >>>> > >>>> The problem is that when virtio_pci_find_capability() reads > >>>> pci_find_capability(dev, PCI_CAP_ID_VNDR), 0 is returned; if repeated, > >>>> it returns a valid number (0x84). Timing seems to matter. pci_cfg_read > >>>> trace shows that that first time read does not reach QEMU but others do > >>>> reach QEMU and return what is expected. > >>>> > >>>> How to debug this, any quick ideas? > >>>> The config space is not a MMIO BAR > >>>> or KVM memory slot or anything like this, right? :) Thanks, > >>> > >>> Depends on the platform. > >>> > >>> E.g. on x86, when using cf8/cfc pair, if guest doesn't > >> > >> > >> Is there an easy way to tell if it is this "cf8/cfc" case? > >> > >> I have these bars, is any of them related to cf8/cfc? Thanks, > >> > >> root@le-dbg:~# (qemu) info mtree -f > >> FlatView #0 > >> AS "memory", root: system > >> AS "cpu-memory-0", root: system > >> Root memory region: system > >> 0000000000000000-00000000ffffffff (prio 0, ram): ppc_spapr.ram kvm > >> 0000200080000000-000020008000002f (prio 0, i/o): msix-table > >> 0000200080000800-0000200080000807 (prio 0, i/o): msix-pba > >> 0000210000000000-0000210000000fff (prio 0, i/o): virtio-pci-common > >> 0000210000001000-0000210000001fff (prio 0, i/o): virtio-pci-isr > >> 0000210000002000-0000210000002fff (prio 0, i/o): virtio-pci-device > >> 0000210000003000-0000210000003fff (prio 0, i/o): virtio-pci-notify > >> > > > > > > No, you want stuff in hw/ppc/spapr_pci.c > > > The problem was with our firmware, fixing that now. > > Out of curiosity. I do not see cf8/cfc on x86 either, or I just do not > recognize those, what is this cf8/cfc? E.g. i440fx: static void i440fx_pcihost_realize(DeviceState *dev, Error **errp) { PCIHostState *s = PCI_HOST_BRIDGE(dev); SysBusDevice *sbd = SYS_BUS_DEVICE(dev); sysbus_add_io(sbd, 0xcf8, &s->conf_mem); sysbus_init_ioports(sbd, 0xcf8, 4); sysbus_add_io(sbd, 0xcfc, &s->data_mem); sysbus_init_ioports(sbd, 0xcfc, 4); /* register i440fx 0xcf8 port as coalesced pio */ memory_region_set_flush_coalesced(&s->data_mem); memory_region_add_coalescing(&s->conf_mem, 0, 4); } > Thanks, > > FlatView #2 > > AS "memory", root: system > > AS "cpu-memory-0", root: system > > AS "piix3-ide", root: bus master container > > AS "virtio-net-pci", root: bus master container > > Root memory region: system > > 0000000000000000-00000000000bffff (prio 0, ram): pc.ram kvm > > 00000000000c0000-00000000000c0fff (prio 0, rom): pc.ram > @00000000000c0000 kvm > 00000000000c1000-00000000000c3fff (prio 0, ram): pc.ram > @00000000000c1000 kvm > 00000000000c4000-00000000000e7fff (prio 0, rom): pc.ram > @00000000000c4000 kvm > 00000000000e8000-00000000000effff (prio 0, ram): pc.ram > @00000000000e8000 kvm > 00000000000f0000-00000000000fffff (prio 0, rom): pc.ram > @00000000000f0000 kvm > 0000000000100000-000000007fffffff (prio 0, ram): pc.ram > @0000000000100000 kvm > 00000000febc0000-00000000febc002f (prio 0, i/o): msix-table > > 00000000febc0800-00000000febc0807 (prio 0, i/o): msix-pba > > 00000000febfc000-00000000febfcfff (prio 0, i/o): virtio-pci-common > > 00000000febfd000-00000000febfdfff (prio 0, i/o): virtio-pci-isr > > 00000000febfe000-00000000febfefff (prio 0, i/o): virtio-pci-device > > 00000000febff000-00000000febfffff (prio 0, i/o): virtio-pci-notify > > 00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic > > 00000000fed00000-00000000fed003ff (prio 0, i/o): hpet > > 00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi > > 00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios kvm > > > > -- > Alexey
diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c index 7abcc50838b8..85b2a7ce96e9 100644 --- a/drivers/virtio/virtio_pci_modern.c +++ b/drivers/virtio/virtio_pci_modern.c @@ -486,9 +486,14 @@ static const struct virtio_config_ops virtio_pci_config_ops = { static inline int virtio_pci_find_capability(struct pci_dev *dev, u8 cfg_type, u32 ioresource_types, int *bars) { - int pos; + int pos = 0;// = pci_find_capability(dev, PCI_CAP_ID_VNDR); - for (pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); + while (!pos) { + pr_err("___K___ (%u) %s %u: FIND a cap\n", smp_processor_id(), __func__, __LINE__); + pos = pci_find_capability(dev, PCI_CAP_ID_VNDR); + pr_err("___K___ (%u) %s %u: cap is at %d\n", smp_processor_id(), __func__, __LINE__, pos);