| Message ID | 20240202191128.1901-5-farosas@suse.de |
|---|---|
| State | New |
| Headers | show |
| Series | migration/multifd: Fix channel creation vs. cleanup races | expand |
On Fri, Feb 02, 2024 at 04:11:27PM -0300, Fabiano Rosas wrote: > We currently have an unfavorable situation around multifd channels > creation and the migration thread execution. > > We create the multifd channels with qio_channel_socket_connect_async > -> qio_task_run_in_thread, but only connect them at the > multifd_new_send_channel_async callback, called from > qio_task_complete, which is registered as a glib event. > > So at multifd_save_setup() we create the channels, but they will only > be actually usable after the whole multifd_save_setup() calling stack > returns back to the main loop. Which means that the migration thread > is already up and running without any possibility for the multifd > channels to be ready on time. > > We currently rely on the channels-ready semaphore blocking > multifd_send_sync_main() until channels start to come up and release > it. However there have been bugs recently found when a channel's > creation fails and multifd_save_cleanup() is allowed to run while > other channels are still being created. > > Let's start to organize this situation by moving the > multifd_save_setup() call into the migration thread. That way we > unblock the main-loop to dispatch the completion callbacks and > actually have a chance of getting the multifd channels ready for when > the migration thread needs them. > > The next patches will deal with the synchronization aspects. > > Note that this takes multifd_save_setup() out of the BQL. > > Signed-off-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com>
diff --git a/migration/migration.c b/migration/migration.c index 55abb175cc..c14d12497f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3315,6 +3315,10 @@ static void *migration_thread(void *opaque) object_ref(OBJECT(s)); update_iteration_initial_status(s); + if (!multifd_save_setup()) { + goto out; + } + bql_lock(); qemu_savevm_state_header(s->to_dst_file); bql_unlock(); @@ -3386,6 +3390,7 @@ static void *migration_thread(void *opaque) urgent = migration_rate_limit(); } +out: trace_migration_thread_after_loop(); migration_iteration_finish(s); object_unref(OBJECT(s)); @@ -3623,11 +3628,6 @@ void migrate_fd_connect(MigrationState *s, Error *error_in) return; } - if (!multifd_save_setup()) { - migrate_fd_cleanup(s); - return; - } - if (migrate_background_snapshot()) { qemu_thread_create(&s->thread, "bg_snapshot", bg_migration_thread, s, QEMU_THREAD_JOINABLE);
We currently have an unfavorable situation around multifd channels creation and the migration thread execution. We create the multifd channels with qio_channel_socket_connect_async -> qio_task_run_in_thread, but only connect them at the multifd_new_send_channel_async callback, called from qio_task_complete, which is registered as a glib event. So at multifd_save_setup() we create the channels, but they will only be actually usable after the whole multifd_save_setup() calling stack returns back to the main loop. Which means that the migration thread is already up and running without any possibility for the multifd channels to be ready on time. We currently rely on the channels-ready semaphore blocking multifd_send_sync_main() until channels start to come up and release it. However there have been bugs recently found when a channel's creation fails and multifd_save_cleanup() is allowed to run while other channels are still being created. Let's start to organize this situation by moving the multifd_save_setup() call into the migration thread. That way we unblock the main-loop to dispatch the completion callbacks and actually have a chance of getting the multifd channels ready for when the migration thread needs them. The next patches will deal with the synchronization aspects. Note that this takes multifd_save_setup() out of the BQL. Signed-off-by: Fabiano Rosas <farosas@suse.de> --- migration/migration.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)