| Message ID | 1697748466-373230-3-git-send-email-steven.sistare@oracle.com |
|---|---|
| State | New |
| Headers | show |
| Series | Live Update reboot mode | expand |
Steve Sistare <steven.sistare@oracle.com> wrote: > Extend the blocker interface so that a blocker can be registered for > one or more migration modes. The existing interfaces register a > blocker for all modes, and the new interfaces take a varargs list > of modes. > > Internally, maintain a separate blocker list per mode. The same Error > object may be added to multiple lists. When a block is deleted, it is > removed from every list, and the Error is freed. > > No functional change until a new mode is added. > > Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Juan Quintela <quintela@redhat.com>
On Thu, Oct 19, 2023 at 01:47:44PM -0700, Steve Sistare wrote: > Extend the blocker interface so that a blocker can be registered for > one or more migration modes. The existing interfaces register a > blocker for all modes, and the new interfaces take a varargs list > of modes. > > Internally, maintain a separate blocker list per mode. The same Error > object may be added to multiple lists. When a block is deleted, it is > removed from every list, and the Error is freed. I'm not sure that assocating blockers with migration modes is the optimal way to model this. IIUC, some of the migration blockers exist because the feature relies on state that only exists on the current host. This isn't a problem with CPR since the migration is within the same host. At the time though, these blockers should likely be redundant for a normal migration that uses "localhost". We can't express the distinction between localhost-migrate and cross-host-migrate historically, but we should have done. This new patch largely enables that I think which is good. What I think this means is that we shouldn't tie blockers to modes, but rather have different types of blockers as a bit set enum MigrationBlockerType { MIGRATION_BLOCKER_LOCAL_HOST = (1 << 0), MIGRATION_BLOCKER_CROSS_HOST = (1 << 1), }; #define MIGRATION_BLOCKER_ALL 0xff Cpr would check for blockers with MIGRATION_BLOCKER_LOCAL_HOST set only. Normal migration within localhost only would similarly only check MIGRATION_BLOCKER_LOCAL_HOST Normal migration between arbitrary host would check for MIGRATION_BLOCKER_LOCAL_HOST and MIGRATION_BLOCKER_CROSS_HOST > > No functional change until a new mode is added. > > Signed-off-by: Steve Sistare <steven.sistare@oracle.com> > --- > include/migration/blocker.h | 44 +++++++++++++++++++-- > migration/migration.c | 95 ++++++++++++++++++++++++++++++++++++++------- > stubs/migr-blocker.c | 10 +++++ > 3 files changed, 132 insertions(+), 17 deletions(-) > > diff --git a/include/migration/blocker.h b/include/migration/blocker.h > index b048f30..a687ac0 100644 > --- a/include/migration/blocker.h > +++ b/include/migration/blocker.h > @@ -14,8 +14,12 @@ > #ifndef MIGRATION_BLOCKER_H > #define MIGRATION_BLOCKER_H > > +#include "qapi/qapi-types-migration.h" > + > +#define MIG_MODE_ALL MIG_MODE__MAX > + > /** > - * @migrate_add_blocker - prevent migration from proceeding > + * @migrate_add_blocker - prevent all modes of migration from proceeding > * > * @reasonp - address of an error to be returned whenever migration is attempted > * > @@ -30,8 +34,8 @@ > int migrate_add_blocker(Error **reasonp, Error **errp); > > /** > - * @migrate_add_blocker_internal - prevent migration from proceeding without > - * only-migrate implications > + * @migrate_add_blocker_internal - prevent all modes of migration from > + * proceeding, but ignore -only-migratable > * > * @reasonp - address of an error to be returned whenever migration is attempted > * > @@ -50,7 +54,7 @@ int migrate_add_blocker(Error **reasonp, Error **errp); > int migrate_add_blocker_internal(Error **reasonp, Error **errp); > > /** > - * @migrate_del_blocker - remove a blocking error from migration and free it. > + * @migrate_del_blocker - remove a migration blocker from all modes and free it. > * > * @reasonp - address of the error blocking migration > * > @@ -58,4 +62,36 @@ int migrate_add_blocker_internal(Error **reasonp, Error **errp); > */ > void migrate_del_blocker(Error **reasonp); > > +/** > + * @migrate_add_blocker_normal - prevent normal migration mode from proceeding > + * > + * @reasonp - address of an error to be returned whenever migration is attempted > + * > + * @errp - [out] The reason (if any) we cannot block migration right now. > + * > + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. > + * > + * *@reasonp is freed and set to NULL if failure is returned. > + * On success, the caller must not free @reasonp, except by > + * calling migrate_del_blocker. > + */ > +int migrate_add_blocker_normal(Error **reasonp, Error **errp); > + > +/** > + * @migrate_add_blocker_modes - prevent some modes of migration from proceeding > + * > + * @reasonp - address of an error to be returned whenever migration is attempted > + * > + * @errp - [out] The reason (if any) we cannot block migration right now. > + * > + * @mode - one or more migration modes to be blocked. The list is terminated > + * by -1 or MIG_MODE_ALL. For the latter, all modes are blocked. > + * > + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. > + * > + * *@reasonp is freed and set to NULL if failure is returned. > + * On success, the caller must not free *@reasonp before the blocker is removed. > + */ > +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...); > + > #endif > diff --git a/migration/migration.c b/migration/migration.c > index 67547eb..b8b54e6 100644 > --- a/migration/migration.c > +++ b/migration/migration.c > @@ -92,7 +92,7 @@ enum mig_rp_message_type { > static MigrationState *current_migration; > static MigrationIncomingState *current_incoming; > > -static GSList *migration_blockers; > +static GSList *migration_blockers[MIG_MODE__MAX]; > > static bool migration_object_check(MigrationState *ms, Error **errp); > static int migration_maybe_pause(MigrationState *s, > @@ -1011,7 +1011,7 @@ static void fill_source_migration_info(MigrationInfo *info) > { > MigrationState *s = migrate_get_current(); > int state = qatomic_read(&s->state); > - GSList *cur_blocker = migration_blockers; > + GSList *cur_blocker = migration_blockers[migrate_mode()]; > > info->blocked_reasons = NULL; > > @@ -1475,38 +1475,105 @@ int migrate_init(MigrationState *s, Error **errp) > return 0; > } > > -int migrate_add_blocker_internal(Error **reasonp, Error **errp) > +static bool is_busy(Error **reasonp, Error **errp) > { > + ERRP_GUARD(); > + > /* Snapshots are similar to migrations, so check RUN_STATE_SAVE_VM too. */ > if (runstate_check(RUN_STATE_SAVE_VM) || !migration_is_idle()) { > error_propagate_prepend(errp, *reasonp, > "disallowing migration blocker " > "(migration/snapshot in progress) for: "); > *reasonp = NULL; > - return -EBUSY; > + return true; > } > - > - migration_blockers = g_slist_prepend(migration_blockers, *reasonp); > - return 0; > + return false; > } > > -int migrate_add_blocker(Error **reasonp, Error **errp) > +static bool is_only_migratable(Error **reasonp, Error **errp, int modes) > { > - if (only_migratable) { > + ERRP_GUARD(); > + > + if (only_migratable && (modes & BIT(MIG_MODE_NORMAL))) { > error_propagate_prepend(errp, *reasonp, > "disallowing migration blocker " > "(--only-migratable) for: "); > *reasonp = NULL; > + return true; > + } > + return false; > +} > + > +static int get_modes(MigMode mode, va_list ap) > +{ > + int modes = 0; > + > + while (mode != -1 && mode != MIG_MODE_ALL) { > + assert(mode >= MIG_MODE_NORMAL && mode < MIG_MODE__MAX); > + modes |= BIT(mode); > + mode = va_arg(ap, MigMode); > + } > + if (mode == MIG_MODE_ALL) { > + modes = BIT(MIG_MODE__MAX) - 1; > + } > + return modes; > +} > + > +static int add_blockers(Error **reasonp, Error **errp, int modes) > +{ > + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { > + if (modes & BIT(mode)) { > + migration_blockers[mode] = g_slist_prepend(migration_blockers[mode], > + *reasonp); > + } > + } > + return 0; > +} > + > +int migrate_add_blocker(Error **reasonp, Error **errp) > +{ > + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_ALL); > +} > + > +int migrate_add_blocker_normal(Error **reasonp, Error **errp) > +{ > + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_NORMAL, -1); > +} > + > +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) > +{ > + int modes; > + va_list ap; > + > + va_start(ap, mode); > + modes = get_modes(mode, ap); > + va_end(ap); > + > + if (is_only_migratable(reasonp, errp, modes)) { > return -EACCES; > + } else if (is_busy(reasonp, errp)) { > + return -EBUSY; > } > + return add_blockers(reasonp, errp, modes); > +} > > - return migrate_add_blocker_internal(reasonp, errp); > +int migrate_add_blocker_internal(Error **reasonp, Error **errp) > +{ > + int modes = BIT(MIG_MODE__MAX) - 1; > + > + if (is_busy(reasonp, errp)) { > + return -EBUSY; > + } > + return add_blockers(reasonp, errp, modes); > } > > void migrate_del_blocker(Error **reasonp) > { > if (*reasonp) { > - migration_blockers = g_slist_remove(migration_blockers, *reasonp); > + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { > + migration_blockers[mode] = g_slist_remove(migration_blockers[mode], > + *reasonp); > + } > error_free(*reasonp); > *reasonp = NULL; > } > @@ -1602,12 +1669,14 @@ void qmp_migrate_pause(Error **errp) > > bool migration_is_blocked(Error **errp) > { > + GSList *blockers = migration_blockers[migrate_mode()]; > + > if (qemu_savevm_state_blocked(errp)) { > return true; > } > > - if (migration_blockers) { > - error_propagate(errp, error_copy(migration_blockers->data)); > + if (blockers) { > + error_propagate(errp, error_copy(blockers->data)); > return true; > } > > diff --git a/stubs/migr-blocker.c b/stubs/migr-blocker.c > index 17a5dbf..11cbff2 100644 > --- a/stubs/migr-blocker.c > +++ b/stubs/migr-blocker.c > @@ -6,6 +6,16 @@ int migrate_add_blocker(Error **reasonp, Error **errp) > return 0; > } > > +int migrate_add_blocker_normal(Error **reasonp, Error **errp) > +{ > + return 0; > +} > + > +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) > +{ > + return 0; > +} > + > void migrate_del_blocker(Error **reasonp) > { > } > -- > 1.8.3.1 > > With regards, Daniel
On 10/23/2023 8:46 AM, Daniel P. Berrangé wrote: > On Thu, Oct 19, 2023 at 01:47:44PM -0700, Steve Sistare wrote: >> Extend the blocker interface so that a blocker can be registered for >> one or more migration modes. The existing interfaces register a >> blocker for all modes, and the new interfaces take a varargs list >> of modes. >> >> Internally, maintain a separate blocker list per mode. The same Error >> object may be added to multiple lists. When a block is deleted, it is >> removed from every list, and the Error is freed. > > I'm not sure that assocating blockers with migration modes is > the optimal way to model this. > > IIUC, some of the migration blockers exist because the feature > relies on state that only exists on the current host. > > This isn't a problem with CPR since the migration is within > the same host. At the time though, these blockers should > likely be redundant for a normal migration that uses "localhost". > > We can't express the distinction between localhost-migrate > and cross-host-migrate historically, but we should have done. > This new patch largely enables that I think which is good. > > What I think this means is that we shouldn't tie blockers > to modes, but rather have different types of blockers as > a bit set > > enum MigrationBlockerType { > MIGRATION_BLOCKER_LOCAL_HOST = (1 << 0), > MIGRATION_BLOCKER_CROSS_HOST = (1 << 1), > }; > > #define MIGRATION_BLOCKER_ALL 0xff > > > Cpr would check for blockers with MIGRATION_BLOCKER_LOCAL_HOST > set only. > > Normal migration within localhost only would similarly only > check MIGRATION_BLOCKER_LOCAL_HOST > > Normal migration between arbitrary host would check for > MIGRATION_BLOCKER_LOCAL_HOST and MIGRATION_BLOCKER_CROSS_HOST Or, we could define MIG_MODE_LOCAL to relax the blockers for local migrations. The user would add mode explicitly to the migrate command, or we could implicitly switch from normal mode to local mode if we infer that the src and target are the same node. MIG_MODE_LOCAL and MIG_MODE_CPR_REBOOT would relax the same blockers for now, but conceivably that could change. When I add cpr-exec mode, it will have its own mode-specific blockers. But, in your scheme, it could map to a new MigrationBlockerType. I do prefer mode as the way of specifying the type of migration. The question is whether we map mode directly to blockers, or map mode plus other criteria such as locality to MigrationBlockerType(s) which map to blockers. One consideration is, how will the user specify the equivalent of only-migratable on the command line? I was thinking of adding -only-migratable <mode1,mode2,...> in a future patch, but if additional criteria maps to blockers, then we need additional options or syntax. - Steve >> No functional change until a new mode is added. >> >> Signed-off-by: Steve Sistare <steven.sistare@oracle.com> >> --- >> include/migration/blocker.h | 44 +++++++++++++++++++-- >> migration/migration.c | 95 ++++++++++++++++++++++++++++++++++++++------- >> stubs/migr-blocker.c | 10 +++++ >> 3 files changed, 132 insertions(+), 17 deletions(-) >> >> diff --git a/include/migration/blocker.h b/include/migration/blocker.h >> index b048f30..a687ac0 100644 >> --- a/include/migration/blocker.h >> +++ b/include/migration/blocker.h >> @@ -14,8 +14,12 @@ >> #ifndef MIGRATION_BLOCKER_H >> #define MIGRATION_BLOCKER_H >> >> +#include "qapi/qapi-types-migration.h" >> + >> +#define MIG_MODE_ALL MIG_MODE__MAX >> + >> /** >> - * @migrate_add_blocker - prevent migration from proceeding >> + * @migrate_add_blocker - prevent all modes of migration from proceeding >> * >> * @reasonp - address of an error to be returned whenever migration is attempted >> * >> @@ -30,8 +34,8 @@ >> int migrate_add_blocker(Error **reasonp, Error **errp); >> >> /** >> - * @migrate_add_blocker_internal - prevent migration from proceeding without >> - * only-migrate implications >> + * @migrate_add_blocker_internal - prevent all modes of migration from >> + * proceeding, but ignore -only-migratable >> * >> * @reasonp - address of an error to be returned whenever migration is attempted >> * >> @@ -50,7 +54,7 @@ int migrate_add_blocker(Error **reasonp, Error **errp); >> int migrate_add_blocker_internal(Error **reasonp, Error **errp); >> >> /** >> - * @migrate_del_blocker - remove a blocking error from migration and free it. >> + * @migrate_del_blocker - remove a migration blocker from all modes and free it. >> * >> * @reasonp - address of the error blocking migration >> * >> @@ -58,4 +62,36 @@ int migrate_add_blocker_internal(Error **reasonp, Error **errp); >> */ >> void migrate_del_blocker(Error **reasonp); >> >> +/** >> + * @migrate_add_blocker_normal - prevent normal migration mode from proceeding >> + * >> + * @reasonp - address of an error to be returned whenever migration is attempted >> + * >> + * @errp - [out] The reason (if any) we cannot block migration right now. >> + * >> + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. >> + * >> + * *@reasonp is freed and set to NULL if failure is returned. >> + * On success, the caller must not free @reasonp, except by >> + * calling migrate_del_blocker. >> + */ >> +int migrate_add_blocker_normal(Error **reasonp, Error **errp); >> + >> +/** >> + * @migrate_add_blocker_modes - prevent some modes of migration from proceeding >> + * >> + * @reasonp - address of an error to be returned whenever migration is attempted >> + * >> + * @errp - [out] The reason (if any) we cannot block migration right now. >> + * >> + * @mode - one or more migration modes to be blocked. The list is terminated >> + * by -1 or MIG_MODE_ALL. For the latter, all modes are blocked. >> + * >> + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. >> + * >> + * *@reasonp is freed and set to NULL if failure is returned. >> + * On success, the caller must not free *@reasonp before the blocker is removed. >> + */ >> +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...); >> + >> #endif >> diff --git a/migration/migration.c b/migration/migration.c >> index 67547eb..b8b54e6 100644 >> --- a/migration/migration.c >> +++ b/migration/migration.c >> @@ -92,7 +92,7 @@ enum mig_rp_message_type { >> static MigrationState *current_migration; >> static MigrationIncomingState *current_incoming; >> >> -static GSList *migration_blockers; >> +static GSList *migration_blockers[MIG_MODE__MAX]; >> >> static bool migration_object_check(MigrationState *ms, Error **errp); >> static int migration_maybe_pause(MigrationState *s, >> @@ -1011,7 +1011,7 @@ static void fill_source_migration_info(MigrationInfo *info) >> { >> MigrationState *s = migrate_get_current(); >> int state = qatomic_read(&s->state); >> - GSList *cur_blocker = migration_blockers; >> + GSList *cur_blocker = migration_blockers[migrate_mode()]; >> >> info->blocked_reasons = NULL; >> >> @@ -1475,38 +1475,105 @@ int migrate_init(MigrationState *s, Error **errp) >> return 0; >> } >> >> -int migrate_add_blocker_internal(Error **reasonp, Error **errp) >> +static bool is_busy(Error **reasonp, Error **errp) >> { >> + ERRP_GUARD(); >> + >> /* Snapshots are similar to migrations, so check RUN_STATE_SAVE_VM too. */ >> if (runstate_check(RUN_STATE_SAVE_VM) || !migration_is_idle()) { >> error_propagate_prepend(errp, *reasonp, >> "disallowing migration blocker " >> "(migration/snapshot in progress) for: "); >> *reasonp = NULL; >> - return -EBUSY; >> + return true; >> } >> - >> - migration_blockers = g_slist_prepend(migration_blockers, *reasonp); >> - return 0; >> + return false; >> } >> >> -int migrate_add_blocker(Error **reasonp, Error **errp) >> +static bool is_only_migratable(Error **reasonp, Error **errp, int modes) >> { >> - if (only_migratable) { >> + ERRP_GUARD(); >> + >> + if (only_migratable && (modes & BIT(MIG_MODE_NORMAL))) { >> error_propagate_prepend(errp, *reasonp, >> "disallowing migration blocker " >> "(--only-migratable) for: "); >> *reasonp = NULL; >> + return true; >> + } >> + return false; >> +} >> + >> +static int get_modes(MigMode mode, va_list ap) >> +{ >> + int modes = 0; >> + >> + while (mode != -1 && mode != MIG_MODE_ALL) { >> + assert(mode >= MIG_MODE_NORMAL && mode < MIG_MODE__MAX); >> + modes |= BIT(mode); >> + mode = va_arg(ap, MigMode); >> + } >> + if (mode == MIG_MODE_ALL) { >> + modes = BIT(MIG_MODE__MAX) - 1; >> + } >> + return modes; >> +} >> + >> +static int add_blockers(Error **reasonp, Error **errp, int modes) >> +{ >> + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { >> + if (modes & BIT(mode)) { >> + migration_blockers[mode] = g_slist_prepend(migration_blockers[mode], >> + *reasonp); >> + } >> + } >> + return 0; >> +} >> + >> +int migrate_add_blocker(Error **reasonp, Error **errp) >> +{ >> + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_ALL); >> +} >> + >> +int migrate_add_blocker_normal(Error **reasonp, Error **errp) >> +{ >> + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_NORMAL, -1); >> +} >> + >> +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) >> +{ >> + int modes; >> + va_list ap; >> + >> + va_start(ap, mode); >> + modes = get_modes(mode, ap); >> + va_end(ap); >> + >> + if (is_only_migratable(reasonp, errp, modes)) { >> return -EACCES; >> + } else if (is_busy(reasonp, errp)) { >> + return -EBUSY; >> } >> + return add_blockers(reasonp, errp, modes); >> +} >> >> - return migrate_add_blocker_internal(reasonp, errp); >> +int migrate_add_blocker_internal(Error **reasonp, Error **errp) >> +{ >> + int modes = BIT(MIG_MODE__MAX) - 1; >> + >> + if (is_busy(reasonp, errp)) { >> + return -EBUSY; >> + } >> + return add_blockers(reasonp, errp, modes); >> } >> >> void migrate_del_blocker(Error **reasonp) >> { >> if (*reasonp) { >> - migration_blockers = g_slist_remove(migration_blockers, *reasonp); >> + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { >> + migration_blockers[mode] = g_slist_remove(migration_blockers[mode], >> + *reasonp); >> + } >> error_free(*reasonp); >> *reasonp = NULL; >> } >> @@ -1602,12 +1669,14 @@ void qmp_migrate_pause(Error **errp) >> >> bool migration_is_blocked(Error **errp) >> { >> + GSList *blockers = migration_blockers[migrate_mode()]; >> + >> if (qemu_savevm_state_blocked(errp)) { >> return true; >> } >> >> - if (migration_blockers) { >> - error_propagate(errp, error_copy(migration_blockers->data)); >> + if (blockers) { >> + error_propagate(errp, error_copy(blockers->data)); >> return true; >> } >> >> diff --git a/stubs/migr-blocker.c b/stubs/migr-blocker.c >> index 17a5dbf..11cbff2 100644 >> --- a/stubs/migr-blocker.c >> +++ b/stubs/migr-blocker.c >> @@ -6,6 +6,16 @@ int migrate_add_blocker(Error **reasonp, Error **errp) >> return 0; >> } >> >> +int migrate_add_blocker_normal(Error **reasonp, Error **errp) >> +{ >> + return 0; >> +} >> + >> +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) >> +{ >> + return 0; >> +} >> + >> void migrate_del_blocker(Error **reasonp) >> { >> } >> -- >> 1.8.3.1 >> >> > > With regards, > Daniel
On Mon, Oct 23, 2023 at 10:37:59AM -0400, Steven Sistare wrote: > On 10/23/2023 8:46 AM, Daniel P. Berrangé wrote: > > On Thu, Oct 19, 2023 at 01:47:44PM -0700, Steve Sistare wrote: > >> Extend the blocker interface so that a blocker can be registered for > >> one or more migration modes. The existing interfaces register a > >> blocker for all modes, and the new interfaces take a varargs list > >> of modes. > >> > >> Internally, maintain a separate blocker list per mode. The same Error > >> object may be added to multiple lists. When a block is deleted, it is > >> removed from every list, and the Error is freed. > > > > I'm not sure that assocating blockers with migration modes is > > the optimal way to model this. > > > > IIUC, some of the migration blockers exist because the feature > > relies on state that only exists on the current host. > > > > This isn't a problem with CPR since the migration is within > > the same host. At the time though, these blockers should > > likely be redundant for a normal migration that uses "localhost". > > > > We can't express the distinction between localhost-migrate > > and cross-host-migrate historically, but we should have done. > > This new patch largely enables that I think which is good. > > > > What I think this means is that we shouldn't tie blockers > > to modes, but rather have different types of blockers as > > a bit set > > > > enum MigrationBlockerType { > > MIGRATION_BLOCKER_LOCAL_HOST = (1 << 0), > > MIGRATION_BLOCKER_CROSS_HOST = (1 << 1), > > }; > > > > #define MIGRATION_BLOCKER_ALL 0xff > > > > > > Cpr would check for blockers with MIGRATION_BLOCKER_LOCAL_HOST > > set only. > > > > Normal migration within localhost only would similarly only > > check MIGRATION_BLOCKER_LOCAL_HOST > > > > Normal migration between arbitrary host would check for > > MIGRATION_BLOCKER_LOCAL_HOST and MIGRATION_BLOCKER_CROSS_HOST > > Or, we could define MIG_MODE_LOCAL to relax the blockers for local migrations. > The user would add mode explicitly to the migrate command, or we could > implicitly switch from normal mode to local mode if we infer that the src > and target are the same node. MIG_MODE_LOCAL and MIG_MODE_CPR_REBOOT would > relax the same blockers for now, but conceivably that could change. > > When I add cpr-exec mode, it will have its own mode-specific blockers. > But, in your scheme, it could map to a new MigrationBlockerType. Yes, there could be further types of blocker. Do you have an example of something that would be a CPR blocker only ? I was thinking that migration blockers have a functional classification which motivates their existance. The different migration modes are describing particular usage scenarios, and a given usage scenario will imply blockers for one or more functional reasons. > I do prefer mode as the way of specifying the type of migration. Sure, I didn't mean to suggest "mode" as an input to 'migrate' is bad. Just that I see migration blockers classification as being distinct from the 'mode'. So a user could specify 'mode' with 'migrate' and that ends up mapping to certain types of blocker. > The question is whether we map mode directly to blockers, or map mode > plus other criteria such as locality to MigrationBlockerType(s) which > map to blockers. > > One consideration is, how will the user specify the equivalent of only-migratable > on the command line? I was thinking of adding -only-migratable <mode1,mode2,...> > in a future patch, but if additional criteria maps to blockers, then we need > additional options or syntax. I guess I could see wanting to use --only-migratable to express that I want a guest that can do a localhost-migration, and CPR, but don't care about cross-host-migration, which would point towards blocker types being exposed. With regards, Daniel
On 10/23/2023 11:02 AM, Daniel P. Berrangé wrote: > On Mon, Oct 23, 2023 at 10:37:59AM -0400, Steven Sistare wrote: >> On 10/23/2023 8:46 AM, Daniel P. Berrangé wrote: >>> On Thu, Oct 19, 2023 at 01:47:44PM -0700, Steve Sistare wrote: >>>> Extend the blocker interface so that a blocker can be registered for >>>> one or more migration modes. The existing interfaces register a >>>> blocker for all modes, and the new interfaces take a varargs list >>>> of modes. >>>> >>>> Internally, maintain a separate blocker list per mode. The same Error >>>> object may be added to multiple lists. When a block is deleted, it is >>>> removed from every list, and the Error is freed. >>> >>> I'm not sure that assocating blockers with migration modes is >>> the optimal way to model this. >>> >>> IIUC, some of the migration blockers exist because the feature >>> relies on state that only exists on the current host. >>> >>> This isn't a problem with CPR since the migration is within >>> the same host. At the time though, these blockers should >>> likely be redundant for a normal migration that uses "localhost". >>> >>> We can't express the distinction between localhost-migrate >>> and cross-host-migrate historically, but we should have done. >>> This new patch largely enables that I think which is good. >>> >>> What I think this means is that we shouldn't tie blockers >>> to modes, but rather have different types of blockers as >>> a bit set >>> >>> enum MigrationBlockerType { >>> MIGRATION_BLOCKER_LOCAL_HOST = (1 << 0), >>> MIGRATION_BLOCKER_CROSS_HOST = (1 << 1), >>> }; >>> >>> #define MIGRATION_BLOCKER_ALL 0xff >>> >>> >>> Cpr would check for blockers with MIGRATION_BLOCKER_LOCAL_HOST >>> set only. >>> >>> Normal migration within localhost only would similarly only >>> check MIGRATION_BLOCKER_LOCAL_HOST >>> >>> Normal migration between arbitrary host would check for >>> MIGRATION_BLOCKER_LOCAL_HOST and MIGRATION_BLOCKER_CROSS_HOST >> >> Or, we could define MIG_MODE_LOCAL to relax the blockers for local migrations. >> The user would add mode explicitly to the migrate command, or we could >> implicitly switch from normal mode to local mode if we infer that the src >> and target are the same node. MIG_MODE_LOCAL and MIG_MODE_CPR_REBOOT would >> relax the same blockers for now, but conceivably that could change. >> >> When I add cpr-exec mode, it will have its own mode-specific blockers. >> But, in your scheme, it could map to a new MigrationBlockerType. > > Yes, there could be further types of blocker. > > Do you have an example of something that would be a CPR blocker > only ? For cpr-exec with vfio, all ram blocks must shared, so the same pinned pages can be attached after exec. Secondary ram blocks, such as vga ram, must be created with memfd. There are misc others. You cannot mix replay and cpr, or colo and cpr. > I was thinking that migration blockers have a functional classification > which motivates their existance. > > The different migration modes are describing particular usage > scenarios, and a given usage scenario will imply blockers for > one or more functional reasons. A "localhost" blocker reason is less useful and less clear-cut than it first seemed. The blockdev blockers that I relaxed for reboot mode must still block normal mode migration to a local host, with concurrent access by the src and target VM's, because they do not support dirty bitmaps. In fact, I'm not sure if any of the blockers would be relaxed for a localhost migration. For cpr, blocks are flushed before qemu exits. >> I do prefer mode as the way of specifying the type of migration. > > Sure, I didn't mean to suggest "mode" as an input to 'migrate' > is bad. Just that I see migration blockers classification as > being distinct from the 'mode'. So a user could specify 'mode' > with 'migrate' and that ends up mapping to certain types of > blocker. > >> The question is whether we map mode directly to blockers, or map mode >> plus other criteria such as locality to MigrationBlockerType(s) which >> map to blockers. >> >> One consideration is, how will the user specify the equivalent of only-migratable >> on the command line? I was thinking of adding -only-migratable <mode1,mode2,...> >> in a future patch, but if additional criteria maps to blockers, then we need >> additional options or syntax. > > I guess I could see wanting to use --only-migratable to express that I > want a guest that can do a localhost-migration, and CPR, but don't > care about cross-host-migration, which would point towards blocker > types being exposed. Yes, but then users need to understand the additional concept of blocker type, and know the mapping between mode and blocker type. I was undecided before, but now I believe that mapping mode to a blocker type does not add much value, and we should stick to blockers based on mode. - Steve
diff --git a/include/migration/blocker.h b/include/migration/blocker.h index b048f30..a687ac0 100644 --- a/include/migration/blocker.h +++ b/include/migration/blocker.h @@ -14,8 +14,12 @@ #ifndef MIGRATION_BLOCKER_H #define MIGRATION_BLOCKER_H +#include "qapi/qapi-types-migration.h" + +#define MIG_MODE_ALL MIG_MODE__MAX + /** - * @migrate_add_blocker - prevent migration from proceeding + * @migrate_add_blocker - prevent all modes of migration from proceeding * * @reasonp - address of an error to be returned whenever migration is attempted * @@ -30,8 +34,8 @@ int migrate_add_blocker(Error **reasonp, Error **errp); /** - * @migrate_add_blocker_internal - prevent migration from proceeding without - * only-migrate implications + * @migrate_add_blocker_internal - prevent all modes of migration from + * proceeding, but ignore -only-migratable * * @reasonp - address of an error to be returned whenever migration is attempted * @@ -50,7 +54,7 @@ int migrate_add_blocker(Error **reasonp, Error **errp); int migrate_add_blocker_internal(Error **reasonp, Error **errp); /** - * @migrate_del_blocker - remove a blocking error from migration and free it. + * @migrate_del_blocker - remove a migration blocker from all modes and free it. * * @reasonp - address of the error blocking migration * @@ -58,4 +62,36 @@ int migrate_add_blocker_internal(Error **reasonp, Error **errp); */ void migrate_del_blocker(Error **reasonp); +/** + * @migrate_add_blocker_normal - prevent normal migration mode from proceeding + * + * @reasonp - address of an error to be returned whenever migration is attempted + * + * @errp - [out] The reason (if any) we cannot block migration right now. + * + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. + * + * *@reasonp is freed and set to NULL if failure is returned. + * On success, the caller must not free @reasonp, except by + * calling migrate_del_blocker. + */ +int migrate_add_blocker_normal(Error **reasonp, Error **errp); + +/** + * @migrate_add_blocker_modes - prevent some modes of migration from proceeding + * + * @reasonp - address of an error to be returned whenever migration is attempted + * + * @errp - [out] The reason (if any) we cannot block migration right now. + * + * @mode - one or more migration modes to be blocked. The list is terminated + * by -1 or MIG_MODE_ALL. For the latter, all modes are blocked. + * + * @returns - 0 on success, -EBUSY/-EACCES on failure, with errp set. + * + * *@reasonp is freed and set to NULL if failure is returned. + * On success, the caller must not free *@reasonp before the blocker is removed. + */ +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...); + #endif diff --git a/migration/migration.c b/migration/migration.c index 67547eb..b8b54e6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -92,7 +92,7 @@ enum mig_rp_message_type { static MigrationState *current_migration; static MigrationIncomingState *current_incoming; -static GSList *migration_blockers; +static GSList *migration_blockers[MIG_MODE__MAX]; static bool migration_object_check(MigrationState *ms, Error **errp); static int migration_maybe_pause(MigrationState *s, @@ -1011,7 +1011,7 @@ static void fill_source_migration_info(MigrationInfo *info) { MigrationState *s = migrate_get_current(); int state = qatomic_read(&s->state); - GSList *cur_blocker = migration_blockers; + GSList *cur_blocker = migration_blockers[migrate_mode()]; info->blocked_reasons = NULL; @@ -1475,38 +1475,105 @@ int migrate_init(MigrationState *s, Error **errp) return 0; } -int migrate_add_blocker_internal(Error **reasonp, Error **errp) +static bool is_busy(Error **reasonp, Error **errp) { + ERRP_GUARD(); + /* Snapshots are similar to migrations, so check RUN_STATE_SAVE_VM too. */ if (runstate_check(RUN_STATE_SAVE_VM) || !migration_is_idle()) { error_propagate_prepend(errp, *reasonp, "disallowing migration blocker " "(migration/snapshot in progress) for: "); *reasonp = NULL; - return -EBUSY; + return true; } - - migration_blockers = g_slist_prepend(migration_blockers, *reasonp); - return 0; + return false; } -int migrate_add_blocker(Error **reasonp, Error **errp) +static bool is_only_migratable(Error **reasonp, Error **errp, int modes) { - if (only_migratable) { + ERRP_GUARD(); + + if (only_migratable && (modes & BIT(MIG_MODE_NORMAL))) { error_propagate_prepend(errp, *reasonp, "disallowing migration blocker " "(--only-migratable) for: "); *reasonp = NULL; + return true; + } + return false; +} + +static int get_modes(MigMode mode, va_list ap) +{ + int modes = 0; + + while (mode != -1 && mode != MIG_MODE_ALL) { + assert(mode >= MIG_MODE_NORMAL && mode < MIG_MODE__MAX); + modes |= BIT(mode); + mode = va_arg(ap, MigMode); + } + if (mode == MIG_MODE_ALL) { + modes = BIT(MIG_MODE__MAX) - 1; + } + return modes; +} + +static int add_blockers(Error **reasonp, Error **errp, int modes) +{ + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { + if (modes & BIT(mode)) { + migration_blockers[mode] = g_slist_prepend(migration_blockers[mode], + *reasonp); + } + } + return 0; +} + +int migrate_add_blocker(Error **reasonp, Error **errp) +{ + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_ALL); +} + +int migrate_add_blocker_normal(Error **reasonp, Error **errp) +{ + return migrate_add_blocker_modes(reasonp, errp, MIG_MODE_NORMAL, -1); +} + +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) +{ + int modes; + va_list ap; + + va_start(ap, mode); + modes = get_modes(mode, ap); + va_end(ap); + + if (is_only_migratable(reasonp, errp, modes)) { return -EACCES; + } else if (is_busy(reasonp, errp)) { + return -EBUSY; } + return add_blockers(reasonp, errp, modes); +} - return migrate_add_blocker_internal(reasonp, errp); +int migrate_add_blocker_internal(Error **reasonp, Error **errp) +{ + int modes = BIT(MIG_MODE__MAX) - 1; + + if (is_busy(reasonp, errp)) { + return -EBUSY; + } + return add_blockers(reasonp, errp, modes); } void migrate_del_blocker(Error **reasonp) { if (*reasonp) { - migration_blockers = g_slist_remove(migration_blockers, *reasonp); + for (MigMode mode = 0; mode < MIG_MODE__MAX; mode++) { + migration_blockers[mode] = g_slist_remove(migration_blockers[mode], + *reasonp); + } error_free(*reasonp); *reasonp = NULL; } @@ -1602,12 +1669,14 @@ void qmp_migrate_pause(Error **errp) bool migration_is_blocked(Error **errp) { + GSList *blockers = migration_blockers[migrate_mode()]; + if (qemu_savevm_state_blocked(errp)) { return true; } - if (migration_blockers) { - error_propagate(errp, error_copy(migration_blockers->data)); + if (blockers) { + error_propagate(errp, error_copy(blockers->data)); return true; } diff --git a/stubs/migr-blocker.c b/stubs/migr-blocker.c index 17a5dbf..11cbff2 100644 --- a/stubs/migr-blocker.c +++ b/stubs/migr-blocker.c @@ -6,6 +6,16 @@ int migrate_add_blocker(Error **reasonp, Error **errp) return 0; } +int migrate_add_blocker_normal(Error **reasonp, Error **errp) +{ + return 0; +} + +int migrate_add_blocker_modes(Error **reasonp, Error **errp, MigMode mode, ...) +{ + return 0; +} + void migrate_del_blocker(Error **reasonp) { }
Extend the blocker interface so that a blocker can be registered for one or more migration modes. The existing interfaces register a blocker for all modes, and the new interfaces take a varargs list of modes. Internally, maintain a separate blocker list per mode. The same Error object may be added to multiple lists. When a block is deleted, it is removed from every list, and the Error is freed. No functional change until a new mode is added. Signed-off-by: Steve Sistare <steven.sistare@oracle.com> --- include/migration/blocker.h | 44 +++++++++++++++++++-- migration/migration.c | 95 ++++++++++++++++++++++++++++++++++++++------- stubs/migr-blocker.c | 10 +++++ 3 files changed, 132 insertions(+), 17 deletions(-)