| Message ID | 20231005092926.56231-1-vsementsov@yandex-team.ru |
|---|---|
| Headers | show |
| Series | pci hotplug tracking | expand |
Ping.
And some addition. We have the case, when the commit
commit 7bed89958bfbf40df9ca681cefbdca63abdde39d
Author: Maxim Levitsky <mlevitsk@redhat.com>
Date: Tue Oct 6 14:38:58 2020 +0200
device_core: use drain_call_rcu in in qmp_device_add
Soon, a device removal might only happen on RCU callback execution.
This is okay for device-del which provides a DEVICE_DELETED event,
but not for the failure case of device-add. To avoid changing
monitor semantics, just drain all pending RCU callbacks on error.
sensibly slows down vm initialization (several calls to device_add of pc-dimm).
And looking at commit message, I see that what I do in the series is exactly a suggestion to change monitor semantics.
What do you think?
Maybe we need a boolean "async" parameter for device_add, which will turn off drain_call_rcu() call and rely on user to handle DEVICE_ON?
On 05.10.23 12:29, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> Main thing this series does is DEVICE_ON event - a counter-part to
> DEVICE_DELETED. A guest-driven event that device is powered-on.
> Details are in patch 2. The new event is paried with corresponding
> command query-hotplug.
>
>
> v8:
> - improve naming, wording and style
> - make new QMP interface experimental
>
>
> Vladimir Sementsov-Ogievskiy (4):
> qapi/qdev.json: unite DEVICE_* event data into single structure
> qapi: add DEVICE_ON and query-hotplug infrastructure
> shpc: implement DEVICE_ON event and query-hotplug
> pcie: implement DEVICE_ON event and query-hotplug
>
> hw/core/hotplug.c | 12 +++
> hw/pci-bridge/pci_bridge_dev.c | 14 +++
> hw/pci-bridge/pcie_pci_bridge.c | 1 +
> hw/pci/pcie.c | 83 +++++++++++++++
> hw/pci/pcie_port.c | 1 +
> hw/pci/shpc.c | 86 +++++++++++++++
> include/hw/hotplug.h | 11 ++
> include/hw/pci/pci_bridge.h | 2 +
> include/hw/pci/pcie.h | 2 +
> include/hw/pci/shpc.h | 2 +
> include/hw/qdev-core.h | 7 ++
> include/monitor/qdev.h | 6 ++
> qapi/qdev.json | 178 +++++++++++++++++++++++++++++---
> softmmu/qdev-monitor.c | 58 +++++++++++
> 14 files changed, 451 insertions(+), 12 deletions(-)
>
[cc Peter, Nikolay and libvirt list] On 02.11.23 11:06, Vladimir Sementsov-Ogievskiy wrote: > Ping. > > And some addition. We have the case, when the commit > > commit 7bed89958bfbf40df9ca681cefbdca63abdde39d > Author: Maxim Levitsky <mlevitsk@redhat.com> > Date: Tue Oct 6 14:38:58 2020 +0200 > > device_core: use drain_call_rcu in in qmp_device_add > Soon, a device removal might only happen on RCU callback execution. > This is okay for device-del which provides a DEVICE_DELETED event, > but not for the failure case of device-add. To avoid changing > monitor semantics, just drain all pending RCU callbacks on error. > > sensibly slows down vm initialization (several calls to device_add of pc-dimm). > > And looking at commit message, I see that what I do in the series is exactly a suggestion to change monitor semantics. > > What do you think? > > Maybe we need a boolean "async" parameter for device_add, which will turn off drain_call_rcu() call and rely on user to handle DEVICE_ON? > > On 05.10.23 12:29, Vladimir Sementsov-Ogievskiy wrote: >> Hi all! >> >> Main thing this series does is DEVICE_ON event - a counter-part to >> DEVICE_DELETED. A guest-driven event that device is powered-on. >> Details are in patch 2. The new event is paried with corresponding >> command query-hotplug. >> >> >> v8: >> - improve naming, wording and style >> - make new QMP interface experimental >> >> >> Vladimir Sementsov-Ogievskiy (4): >> qapi/qdev.json: unite DEVICE_* event data into single structure >> qapi: add DEVICE_ON and query-hotplug infrastructure >> shpc: implement DEVICE_ON event and query-hotplug >> pcie: implement DEVICE_ON event and query-hotplug >> >> hw/core/hotplug.c | 12 +++ >> hw/pci-bridge/pci_bridge_dev.c | 14 +++ >> hw/pci-bridge/pcie_pci_bridge.c | 1 + >> hw/pci/pcie.c | 83 +++++++++++++++ >> hw/pci/pcie_port.c | 1 + >> hw/pci/shpc.c | 86 +++++++++++++++ >> include/hw/hotplug.h | 11 ++ >> include/hw/pci/pci_bridge.h | 2 + >> include/hw/pci/pcie.h | 2 + >> include/hw/pci/shpc.h | 2 + >> include/hw/qdev-core.h | 7 ++ >> include/monitor/qdev.h | 6 ++ >> qapi/qdev.json | 178 +++++++++++++++++++++++++++++--- >> softmmu/qdev-monitor.c | 58 +++++++++++ >> 14 files changed, 451 insertions(+), 12 deletions(-) >> >
On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > Main thing this series does is DEVICE_ON event - a counter-part to > DEVICE_DELETED. A guest-driven event that device is powered-on. > Details are in patch 2. The new event is paried with corresponding > command query-hotplug. Several things questionable here: 1. depending on guest activity you can get as many DEVICE_ON events as you like 2. it's just for shpc and native pcie - things are confusing enough for management, we should make sure it can work for all devices 3. what about non hotpluggable devices? do we want the event for them? I feel this needs actual motivation so we can judge what's the right way to do it. > > v8: > - improve naming, wording and style > - make new QMP interface experimental > > > Vladimir Sementsov-Ogievskiy (4): > qapi/qdev.json: unite DEVICE_* event data into single structure > qapi: add DEVICE_ON and query-hotplug infrastructure > shpc: implement DEVICE_ON event and query-hotplug > pcie: implement DEVICE_ON event and query-hotplug > > hw/core/hotplug.c | 12 +++ > hw/pci-bridge/pci_bridge_dev.c | 14 +++ > hw/pci-bridge/pcie_pci_bridge.c | 1 + > hw/pci/pcie.c | 83 +++++++++++++++ > hw/pci/pcie_port.c | 1 + > hw/pci/shpc.c | 86 +++++++++++++++ > include/hw/hotplug.h | 11 ++ > include/hw/pci/pci_bridge.h | 2 + > include/hw/pci/pcie.h | 2 + > include/hw/pci/shpc.h | 2 + > include/hw/qdev-core.h | 7 ++ > include/monitor/qdev.h | 6 ++ > qapi/qdev.json | 178 +++++++++++++++++++++++++++++--- > softmmu/qdev-monitor.c | 58 +++++++++++ > 14 files changed, 451 insertions(+), 12 deletions(-) > > -- > 2.34.1
On 02.11.23 14:31, Michael S. Tsirkin wrote: > On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: >> Hi all! >> >> Main thing this series does is DEVICE_ON event - a counter-part to >> DEVICE_DELETED. A guest-driven event that device is powered-on. >> Details are in patch 2. The new event is paried with corresponding >> command query-hotplug. > > Several things questionable here: > 1. depending on guest activity you can get as many > DEVICE_ON events as you like No, I've made it so it may be sent only once per device > 2. it's just for shpc and native pcie - things are > confusing enough for management, we should make sure > it can work for all devices Agree, I'm thinking about it > 3. what about non hotpluggable devices? do we want the event for them? > I think, yes, especially if we make async=true|false flag for device_add, so that successful device_add must be always followed by DEVICE_ON - like device_del is followed by DEVICE_DELETED. Maybe, to generalize, it should be called not DEVICE_ON (which mostly relate to hotplug controller statuses) but DEVICE_ADDED - a full counterpart for DEVICE_DELETED. > > I feel this needs actual motivation so we can judge what's the > right way to do it. My first motivation for this series was the fact that successful device_add doesn't guarantee that hard disk successfully hotplugged to the guest. It relates to some problems with shpc/pcie hotplug we had in the past, and they are mostly fixed. But still, for management tool it's good to understand that all actions related to hotplug controller are done and we have "green light". Recently new motivation come, as I described in my "ping" letter <6bd19a07-5224-464d-b54d-1d738f5ba8f7@yandex-team.ru>, that we have a performance degradation because of 7bed89958bfbf40df, which introduces drain_call_rcu() in device_add, to make it more synchronous. So, my suggestion is make it instead more asynchronous (probably with special flag) and rely on DEVICE_ON event. > > >> >> v8: >> - improve naming, wording and style >> - make new QMP interface experimental >> >> >> Vladimir Sementsov-Ogievskiy (4): >> qapi/qdev.json: unite DEVICE_* event data into single structure >> qapi: add DEVICE_ON and query-hotplug infrastructure >> shpc: implement DEVICE_ON event and query-hotplug >> pcie: implement DEVICE_ON event and query-hotplug >> >> hw/core/hotplug.c | 12 +++ >> hw/pci-bridge/pci_bridge_dev.c | 14 +++ >> hw/pci-bridge/pcie_pci_bridge.c | 1 + >> hw/pci/pcie.c | 83 +++++++++++++++ >> hw/pci/pcie_port.c | 1 + >> hw/pci/shpc.c | 86 +++++++++++++++ >> include/hw/hotplug.h | 11 ++ >> include/hw/pci/pci_bridge.h | 2 + >> include/hw/pci/pcie.h | 2 + >> include/hw/pci/shpc.h | 2 + >> include/hw/qdev-core.h | 7 ++ >> include/monitor/qdev.h | 6 ++ >> qapi/qdev.json | 178 +++++++++++++++++++++++++++++--- >> softmmu/qdev-monitor.c | 58 +++++++++++ >> 14 files changed, 451 insertions(+), 12 deletions(-) >> >> -- >> 2.34.1 >
On Thu, Nov 02, 2023 at 03:00:01PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 02.11.23 14:31, Michael S. Tsirkin wrote: > > On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > Hi all! > > > > > > Main thing this series does is DEVICE_ON event - a counter-part to > > > DEVICE_DELETED. A guest-driven event that device is powered-on. > > > Details are in patch 2. The new event is paried with corresponding > > > command query-hotplug. > > > > Several things questionable here: > > 1. depending on guest activity you can get as many > > DEVICE_ON events as you like > > No, I've made it so it may be sent only once per device Maybe document that? > > 2. it's just for shpc and native pcie - things are > > confusing enough for management, we should make sure > > it can work for all devices > > Agree, I'm thinking about it > > > 3. what about non hotpluggable devices? do we want the event for them? > > > > I think, yes, especially if we make async=true|false flag for device_add, so that successful device_add must be always followed by DEVICE_ON - like device_del is followed by DEVICE_DELETED. > > Maybe, to generalize, it should be called not DEVICE_ON (which mostly relate to hotplug controller statuses) but DEVICE_ADDED - a full counterpart for DEVICE_DELETED. > > > > > I feel this needs actual motivation so we can judge what's the > > right way to do it. > > My first motivation for this series was the fact that successful device_add doesn't guarantee that hard disk successfully hotplugged to the guest. It relates to some problems with shpc/pcie hotplug we had in the past, and they are mostly fixed. But still, for management tool it's good to understand that all actions related to hotplug controller are done and we have "green light". what does "successfully" mean though? E.g. a bunch of guests will not properly show you the device if the disk is not formatted properly. > > Recently new motivation come, as I described in my "ping" letter <6bd19a07-5224-464d-b54d-1d738f5ba8f7@yandex-team.ru>, that we have a performance degradation because of 7bed89958bfbf40df, which introduces drain_call_rcu() in device_add, to make it more synchronous. So, my suggestion is make it instead more asynchronous (probably with special flag) and rely on DEVICE_ON event. This one? commit 7bed89958bfbf40df9ca681cefbdca63abdde39d Author: Maxim Levitsky <mlevitsk@redhat.com> Date: Tue Oct 6 14:38:58 2020 +0200 device_core: use drain_call_rcu in in qmp_device_add Soon, a device removal might only happen on RCU callback execution. This is okay for device-del which provides a DEVICE_DELETED event, but not for the failure case of device-add. To avoid changing monitor semantics, just drain all pending RCU callbacks on error. Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com> [Don't use it in qmp_device_del. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c index e9b7228480..bcfb90a08f 100644 --- a/softmmu/qdev-monitor.c +++ b/softmmu/qdev-monitor.c @@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, Error **errp) return; } dev = qdev_device_add(opts, errp); + + /* + * Drain all pending RCU callbacks. This is done because + * some bus related operations can delay a device removal + * (in this case this can happen if device is added and then + * removed due to a configuration error) + * to a RCU callback, but user might expect that this interface + * will finish its job completely once qmp command returns result + * to the user + */ + drain_call_rcu(); + if (!dev) { qemu_opts_del(opts); return; So maybe just move drain_call_rcu under if (!dev) then and be done with it?
On 02.11.23 15:12, Michael S. Tsirkin wrote: > On Thu, Nov 02, 2023 at 03:00:01PM +0300, Vladimir Sementsov-Ogievskiy wrote: >> On 02.11.23 14:31, Michael S. Tsirkin wrote: >>> On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: >>>> Hi all! >>>> >>>> Main thing this series does is DEVICE_ON event - a counter-part to >>>> DEVICE_DELETED. A guest-driven event that device is powered-on. >>>> Details are in patch 2. The new event is paried with corresponding >>>> command query-hotplug. >>> >>> Several things questionable here: >>> 1. depending on guest activity you can get as many >>> DEVICE_ON events as you like >> >> No, I've made it so it may be sent only once per device > > Maybe document that? Right, my fault > >>> 2. it's just for shpc and native pcie - things are >>> confusing enough for management, we should make sure >>> it can work for all devices >> >> Agree, I'm thinking about it >> >>> 3. what about non hotpluggable devices? do we want the event for them? >>> >> >> I think, yes, especially if we make async=true|false flag for device_add, so that successful device_add must be always followed by DEVICE_ON - like device_del is followed by DEVICE_DELETED. >> >> Maybe, to generalize, it should be called not DEVICE_ON (which mostly relate to hotplug controller statuses) but DEVICE_ADDED - a full counterpart for DEVICE_DELETED. >> >>> >>> I feel this needs actual motivation so we can judge what's the >>> right way to do it. >> >> My first motivation for this series was the fact that successful device_add doesn't guarantee that hard disk successfully hotplugged to the guest. It relates to some problems with shpc/pcie hotplug we had in the past, and they are mostly fixed. But still, for management tool it's good to understand that all actions related to hotplug controller are done and we have "green light". > > what does "successfully" mean though? E.g. a bunch of guests will not > properly show you the device if the disk is not formatted properly. Yes, I understand, that we may say only about "some degree of success". But here is some physical sense still: DEVICE_ON indicates, that it's now safe to call device_del. And calling device_del before DEVICE_ON is a kind of unexpected behavior. > >> >> Recently new motivation come, as I described in my "ping" letter <6bd19a07-5224-464d-b54d-1d738f5ba8f7@yandex-team.ru>, that we have a performance degradation because of 7bed89958bfbf40df, which introduces drain_call_rcu() in device_add, to make it more synchronous. So, my suggestion is make it instead more asynchronous (probably with special flag) and rely on DEVICE_ON event. > > This one? > > commit 7bed89958bfbf40df9ca681cefbdca63abdde39d > Author: Maxim Levitsky <mlevitsk@redhat.com> > Date: Tue Oct 6 14:38:58 2020 +0200 > > device_core: use drain_call_rcu in in qmp_device_add > > Soon, a device removal might only happen on RCU callback execution. > This is okay for device-del which provides a DEVICE_DELETED event, > but not for the failure case of device-add. To avoid changing > monitor semantics, just drain all pending RCU callbacks on error. > > Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> > Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com> > [Don't use it in qmp_device_del. - Paolo] > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > > diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c > index e9b7228480..bcfb90a08f 100644 > --- a/softmmu/qdev-monitor.c > +++ b/softmmu/qdev-monitor.c > @@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, Error **errp) > return; > } > dev = qdev_device_add(opts, errp); > + > + /* > + * Drain all pending RCU callbacks. This is done because > + * some bus related operations can delay a device removal > + * (in this case this can happen if device is added and then > + * removed due to a configuration error) > + * to a RCU callback, but user might expect that this interface > + * will finish its job completely once qmp command returns result > + * to the user > + */ > + drain_call_rcu(); > + > if (!dev) { > qemu_opts_del(opts); > return; > > > > So maybe just move drain_call_rcu under if (!dev) then and be done with > it? > Hmm, I read the commit message thinking that it saying about device removal by mistake and actually want to say both about device_add and device_del.. But I was wrong? Hmm, it directly say "just drain all pending RCU callbacks on error", but does that on success path as well. Yes, moving drain_call_rcu makes sense for me, and will close the second "motivation". I can make a patch.
On Thu, Nov 02, 2023 at 04:28:43PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 02.11.23 15:12, Michael S. Tsirkin wrote: > > On Thu, Nov 02, 2023 at 03:00:01PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > On 02.11.23 14:31, Michael S. Tsirkin wrote: > > > > On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > > > Hi all! > > > > > > > > > > Main thing this series does is DEVICE_ON event - a counter-part to > > > > > DEVICE_DELETED. A guest-driven event that device is powered-on. > > > > > Details are in patch 2. The new event is paried with corresponding > > > > > command query-hotplug. > > > > > > > > Several things questionable here: > > > > 1. depending on guest activity you can get as many > > > > DEVICE_ON events as you like > > > > > > No, I've made it so it may be sent only once per device > > > > Maybe document that? > > Right, my fault > > > > > > > 2. it's just for shpc and native pcie - things are > > > > confusing enough for management, we should make sure > > > > it can work for all devices > > > > > > Agree, I'm thinking about it > > > > > > > 3. what about non hotpluggable devices? do we want the event for them? > > > > > > > > > > I think, yes, especially if we make async=true|false flag for device_add, so that successful device_add must be always followed by DEVICE_ON - like device_del is followed by DEVICE_DELETED. > > > > > > Maybe, to generalize, it should be called not DEVICE_ON (which mostly relate to hotplug controller statuses) but DEVICE_ADDED - a full counterpart for DEVICE_DELETED. > > > > > > > > > > > I feel this needs actual motivation so we can judge what's the > > > > right way to do it. > > > > > > My first motivation for this series was the fact that successful device_add doesn't guarantee that hard disk successfully hotplugged to the guest. It relates to some problems with shpc/pcie hotplug we had in the past, and they are mostly fixed. But still, for management tool it's good to understand that all actions related to hotplug controller are done and we have "green light". > > > > what does "successfully" mean though? E.g. a bunch of guests will not > > properly show you the device if the disk is not formatted properly. > > Yes, I understand, that we may say only about "some degree of success". > > But here is some physical sense still: DEVICE_ON indicates, that it's now safe to call device_del. And calling device_del before DEVICE_ON is a kind of unexpected behavior. > Is that really true? I really don't think we should introduce new types of undefined behavior. > > > > > > > > Recently new motivation come, as I described in my "ping" letter <6bd19a07-5224-464d-b54d-1d738f5ba8f7@yandex-team.ru>, that we have a performance degradation because of 7bed89958bfbf40df, which introduces drain_call_rcu() in device_add, to make it more synchronous. So, my suggestion is make it instead more asynchronous (probably with special flag) and rely on DEVICE_ON event. > > > > This one? > > > > commit 7bed89958bfbf40df9ca681cefbdca63abdde39d > > Author: Maxim Levitsky <mlevitsk@redhat.com> > > Date: Tue Oct 6 14:38:58 2020 +0200 > > > > device_core: use drain_call_rcu in in qmp_device_add > > Soon, a device removal might only happen on RCU callback execution. > > This is okay for device-del which provides a DEVICE_DELETED event, > > but not for the failure case of device-add. To avoid changing > > monitor semantics, just drain all pending RCU callbacks on error. > > Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> > > Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > > Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com> > > [Don't use it in qmp_device_del. - Paolo] > > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > > > > diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c > > index e9b7228480..bcfb90a08f 100644 > > --- a/softmmu/qdev-monitor.c > > +++ b/softmmu/qdev-monitor.c > > @@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, Error **errp) > > return; > > } > > dev = qdev_device_add(opts, errp); > > + > > + /* > > + * Drain all pending RCU callbacks. This is done because > > + * some bus related operations can delay a device removal > > + * (in this case this can happen if device is added and then > > + * removed due to a configuration error) > > + * to a RCU callback, but user might expect that this interface > > + * will finish its job completely once qmp command returns result > > + * to the user > > + */ > > + drain_call_rcu(); > > + > > if (!dev) { > > qemu_opts_del(opts); > > return; > > > > > > > > So maybe just move drain_call_rcu under if (!dev) then and be done with > > it? > > > > Hmm, I read the commit message thinking that it saying about device removal by mistake and actually want to say both about device_add and device_del.. But I was wrong? > > Hmm, it directly say "just drain all pending RCU callbacks on error", but does that on success path as well. > > Yes, moving drain_call_rcu makes sense for me, and will close the second "motivation". I can make a patch. > > -- > Best regards, > Vladimir
On 02.11.23 16:32, Michael S. Tsirkin wrote: > On Thu, Nov 02, 2023 at 04:28:43PM +0300, Vladimir Sementsov-Ogievskiy wrote: >> On 02.11.23 15:12, Michael S. Tsirkin wrote: >>> On Thu, Nov 02, 2023 at 03:00:01PM +0300, Vladimir Sementsov-Ogievskiy wrote: >>>> On 02.11.23 14:31, Michael S. Tsirkin wrote: >>>>> On Thu, Oct 05, 2023 at 12:29:22PM +0300, Vladimir Sementsov-Ogievskiy wrote: >>>>>> Hi all! >>>>>> >>>>>> Main thing this series does is DEVICE_ON event - a counter-part to >>>>>> DEVICE_DELETED. A guest-driven event that device is powered-on. >>>>>> Details are in patch 2. The new event is paried with corresponding >>>>>> command query-hotplug. >>>>> >>>>> Several things questionable here: >>>>> 1. depending on guest activity you can get as many >>>>> DEVICE_ON events as you like >>>> >>>> No, I've made it so it may be sent only once per device >>> >>> Maybe document that? >> >> Right, my fault >> >>> >>>>> 2. it's just for shpc and native pcie - things are >>>>> confusing enough for management, we should make sure >>>>> it can work for all devices >>>> >>>> Agree, I'm thinking about it >>>> >>>>> 3. what about non hotpluggable devices? do we want the event for them? >>>>> >>>> >>>> I think, yes, especially if we make async=true|false flag for device_add, so that successful device_add must be always followed by DEVICE_ON - like device_del is followed by DEVICE_DELETED. >>>> >>>> Maybe, to generalize, it should be called not DEVICE_ON (which mostly relate to hotplug controller statuses) but DEVICE_ADDED - a full counterpart for DEVICE_DELETED. >>>> >>>>> >>>>> I feel this needs actual motivation so we can judge what's the >>>>> right way to do it. >>>> >>>> My first motivation for this series was the fact that successful device_add doesn't guarantee that hard disk successfully hotplugged to the guest. It relates to some problems with shpc/pcie hotplug we had in the past, and they are mostly fixed. But still, for management tool it's good to understand that all actions related to hotplug controller are done and we have "green light". >>> >>> what does "successfully" mean though? E.g. a bunch of guests will not >>> properly show you the device if the disk is not formatted properly. >> >> Yes, I understand, that we may say only about "some degree of success". >> >> But here is some physical sense still: DEVICE_ON indicates, that it's now safe to call device_del. And calling device_del before DEVICE_ON is a kind of unexpected behavior. >> > > Is that really true? I really don't think we should introduce new types > of undefined behavior. Good question. At least, it was true before some recent fixes to SHPC.. And even if it is so now, we could instead fix it by some "internal DEVICE_ON", and postpone device deletion until we have it... OK, I need a round to rethink this all. Thanks! > > >>> >>>> >>>> Recently new motivation come, as I described in my "ping" letter <6bd19a07-5224-464d-b54d-1d738f5ba8f7@yandex-team.ru>, that we have a performance degradation because of 7bed89958bfbf40df, which introduces drain_call_rcu() in device_add, to make it more synchronous. So, my suggestion is make it instead more asynchronous (probably with special flag) and rely on DEVICE_ON event. >>> >>> This one? >>> >>> commit 7bed89958bfbf40df9ca681cefbdca63abdde39d >>> Author: Maxim Levitsky <mlevitsk@redhat.com> >>> Date: Tue Oct 6 14:38:58 2020 +0200 >>> >>> device_core: use drain_call_rcu in in qmp_device_add >>> Soon, a device removal might only happen on RCU callback execution. >>> This is okay for device-del which provides a DEVICE_DELETED event, >>> but not for the failure case of device-add. To avoid changing >>> monitor semantics, just drain all pending RCU callbacks on error. >>> Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> >>> Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> >>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> >>> Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com> >>> [Don't use it in qmp_device_del. - Paolo] >>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> >>> >>> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c >>> index e9b7228480..bcfb90a08f 100644 >>> --- a/softmmu/qdev-monitor.c >>> +++ b/softmmu/qdev-monitor.c >>> @@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, Error **errp) >>> return; >>> } >>> dev = qdev_device_add(opts, errp); >>> + >>> + /* >>> + * Drain all pending RCU callbacks. This is done because >>> + * some bus related operations can delay a device removal >>> + * (in this case this can happen if device is added and then >>> + * removed due to a configuration error) >>> + * to a RCU callback, but user might expect that this interface >>> + * will finish its job completely once qmp command returns result >>> + * to the user >>> + */ >>> + drain_call_rcu(); >>> + >>> if (!dev) { >>> qemu_opts_del(opts); >>> return; >>> >>> >>> >>> So maybe just move drain_call_rcu under if (!dev) then and be done with >>> it? >>> >> >> Hmm, I read the commit message thinking that it saying about device removal by mistake and actually want to say both about device_add and device_del.. But I was wrong? >> >> Hmm, it directly say "just drain all pending RCU callbacks on error", but does that on success path as well. >> >> Yes, moving drain_call_rcu makes sense for me, and will close the second "motivation". I can make a patch. >> >> -- >> Best regards, >> Vladimir >