Message ID | 20230320105404.637661-1-berrange@redhat.com |
---|---|
State | New |
Headers | show |
Series | python: honour message limit when using pre-opened socket | expand |
On Mon, Mar 20, 2023 at 2:54 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > The default message recv limit in asyncio is smaller than our needs, so > when opening connections we override it. This was done when opening a > connection using a socket address, but was missed when using a > pre-opened socket file descriptor. > > This latent bug was exposed when the QEMUMachine class was enhanced to > use socketpair() when no socket address was passed by: > > commit bd4c0ef409140bd1be393407c04005ac077d4574 > Author: Marc-André Lureau <marcandre.lureau@redhat.com> > Date: Wed Jan 11 12:01:01 2023 +0400 > > python/qemu/machine: use socketpair() for QMP by default > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > thanks Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com> > --- > python/qemu/qmp/protocol.py | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py > index 22e60298d2..2bd341a93e 100644 > --- a/python/qemu/qmp/protocol.py > +++ b/python/qemu/qmp/protocol.py > @@ -307,7 +307,10 @@ async def open_with_socket(self, sock: socket.socket) > -> None: > > :raise StateError: When the `Runstate` is not `IDLE`. > """ > - self._reader, self._writer = await > asyncio.open_connection(sock=sock) > + self._reader, self._writer = await asyncio.open_connection( > + sock=sock, > + limit=self._limit, > + ) > self._set_state(Runstate.CONNECTING) > > @upper_half > -- > 2.39.2 > >
On 20.03.23 13:54, Daniel P. Berrangé wrote: > The default message recv limit in asyncio is smaller than our needs, so > when opening connections we override it. This was done when opening a > connection using a socket address, but was missed when using a > pre-opened socket file descriptor. > > This latent bug was exposed when the QEMUMachine class was enhanced to > use socketpair() when no socket address was passed by: > > commit bd4c0ef409140bd1be393407c04005ac077d4574 > Author: Marc-André Lureau<marcandre.lureau@redhat.com> > Date: Wed Jan 11 12:01:01 2023 +0400 > > python/qemu/machine: use socketpair() for QMP by default > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com> Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Thanks!
On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> wrote: > > On 20.03.23 13:54, Daniel P. Berrangé wrote: > > The default message recv limit in asyncio is smaller than our needs, so > > when opening connections we override it. This was done when opening a > > connection using a socket address, but was missed when using a > > pre-opened socket file descriptor. > > > > This latent bug was exposed when the QEMUMachine class was enhanced to > > use socketpair() when no socket address was passed by: > > > > commit bd4c0ef409140bd1be393407c04005ac077d4574 > > Author: Marc-André Lureau<marcandre.lureau@redhat.com> > > Date: Wed Jan 11 12:01:01 2023 +0400 > > > > python/qemu/machine: use socketpair() for QMP by default > > > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com> > > Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> > > Thanks! > > -- > Best regards, > Vladimir > Thanks x3. Will stage in both places. --js
On Mon, Mar 20, 2023 at 8:27 PM John Snow <jsnow@redhat.com> wrote: > > On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy > <vsementsov@yandex-team.ru> wrote: > > > > On 20.03.23 13:54, Daniel P. Berrangé wrote: > > > The default message recv limit in asyncio is smaller than our needs, so > > > when opening connections we override it. This was done when opening a > > > connection using a socket address, but was missed when using a > > > pre-opened socket file descriptor. > > > > > > This latent bug was exposed when the QEMUMachine class was enhanced to > > > use socketpair() when no socket address was passed by: > > > > > > commit bd4c0ef409140bd1be393407c04005ac077d4574 > > > Author: Marc-André Lureau<marcandre.lureau@redhat.com> > > > Date: Wed Jan 11 12:01:01 2023 +0400 > > > > > > python/qemu/machine: use socketpair() for QMP by default > > > > > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com> > > > > Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> > > > > Thanks! > > > > -- > > Best regards, > > Vladimir > > > > Thanks x3. Will stage in both places. > ... Realizing I let the two codebases diverge a bit more than I had meant to allow, and the "open with socket" rewrite I made never got brought back to qemu.git, tch. I believe this bug only exists in qemu.git because of that. I need to backport my changes to qemu.git and ensure that there isn't a problem with the revised code as it exists in qemu.qmp. I'll have to fix that *before* I drop qemu.qmp from the qemu.git repo, which I am hoping will be soon after Paolo takes the new Python patches that create a venv during configure time. --js
On Mon, May 15, 2023 at 5:14 PM John Snow <jsnow@redhat.com> wrote: > > On Mon, Mar 20, 2023 at 8:27 PM John Snow <jsnow@redhat.com> wrote: > > > > On Mon, Mar 20, 2023 at 8:20 AM Vladimir Sementsov-Ogievskiy > > <vsementsov@yandex-team.ru> wrote: > > > > > > On 20.03.23 13:54, Daniel P. Berrangé wrote: > > > > The default message recv limit in asyncio is smaller than our needs, so > > > > when opening connections we override it. This was done when opening a > > > > connection using a socket address, but was missed when using a > > > > pre-opened socket file descriptor. > > > > > > > > This latent bug was exposed when the QEMUMachine class was enhanced to > > > > use socketpair() when no socket address was passed by: > > > > > > > > commit bd4c0ef409140bd1be393407c04005ac077d4574 > > > > Author: Marc-André Lureau<marcandre.lureau@redhat.com> > > > > Date: Wed Jan 11 12:01:01 2023 +0400 > > > > > > > > python/qemu/machine: use socketpair() for QMP by default > > > > > > > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com> > > > > > > Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> > > > > > > Thanks! > > > > > > -- > > > Best regards, > > > Vladimir > > > > > > > Thanks x3. Will stage in both places. > > > > ... Realizing I let the two codebases diverge a bit more than I had > meant to allow, and the "open with socket" rewrite I made never got > brought back to qemu.git, tch. I believe this bug only exists in > qemu.git because of that. I need to backport my changes to qemu.git > and ensure that there isn't a problem with the revised code as it > exists in qemu.qmp. > > I'll have to fix that *before* I drop qemu.qmp from the qemu.git repo, > which I am hoping will be soon after Paolo takes the new Python > patches that create a venv during configure time. > > --js OK, I've just sent two series: (1) A MR for python-qemu-qmp here, which backports a few misc fixes from qemu.git back over to python-qemu-qmp; https://gitlab.com/qemu-project/python-qemu-qmp/-/merge_requests/23 This needs approval from at least one person. If you're interested in becoming a "reviewer" there, let me know and I'll add you if you aren't already. (2) A patchset for qemu.git that backports the socket changes that went into python-qemu-qmp. https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04285.html This should fix the bug described here in this series, and is necessary to merge before I swap the in-tree qemu.git for a vendored qemu.qmp package. Review welcomed, but any changes we need will need to happen on the python-qemu-qmp repo first. Dropping the in-tree qemu.qmp package is going to happen after the large series Paolo just sent a PR for, https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04330.html which adds a configure-time venv to the QEMU build system. Hoping to drop qemu.qmp *this dev cycle* and release v0.0.3 for qemu.qmp and vendor it to qemu.git as a .whl file. Thanks, --js
diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py index 22e60298d2..2bd341a93e 100644 --- a/python/qemu/qmp/protocol.py +++ b/python/qemu/qmp/protocol.py @@ -307,7 +307,10 @@ async def open_with_socket(self, sock: socket.socket) -> None: :raise StateError: When the `Runstate` is not `IDLE`. """ - self._reader, self._writer = await asyncio.open_connection(sock=sock) + self._reader, self._writer = await asyncio.open_connection( + sock=sock, + limit=self._limit, + ) self._set_state(Runstate.CONNECTING) @upper_half
The default message recv limit in asyncio is smaller than our needs, so when opening connections we override it. This was done when opening a connection using a socket address, but was missed when using a pre-opened socket file descriptor. This latent bug was exposed when the QEMUMachine class was enhanced to use socketpair() when no socket address was passed by: commit bd4c0ef409140bd1be393407c04005ac077d4574 Author: Marc-André Lureau <marcandre.lureau@redhat.com> Date: Wed Jan 11 12:01:01 2023 +0400 python/qemu/machine: use socketpair() for QMP by default Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- python/qemu/qmp/protocol.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)