| Message ID | 55dc807fe7f2c8989d267c70b50c5af5c0b22f00.camel@linux.ibm.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | powerpc/pseries/vas: VAS/NXGZIP support with LPM | expand |
Hi Haren, Mostly this seems OK to me. Some questions: Haren Myneni <haren@linux.ibm.com> writes: > Since the VAS windows belong to the VAS hardware resource, the > hypervisor expects the partition to close them on source partition > and reopen them after the partition migrated on the destination > machine. Not clear to me what "expects" really means here. Would it be accurate to say "requires" instead? If the OS fails to close the windows before suspend, what happens? > This handler is called before pseries_suspend() to close these > windows and again invoked after migration. All active windows > for both default and QoS types will be closed and mark them > in-active and reopened after migration with this handler. > During the migration, the user space receives paste instruction > failure if it issues copy/paste on these in-active windows. OK, I assume that's tolerable to existing users, is that correct? I.e. users should already be prepared to incur arbitrary paste instruction failures. > > Signed-off-by: Haren Myneni <haren@linux.ibm.com> > --- > arch/powerpc/platforms/pseries/mobility.c | 5 ++ > arch/powerpc/platforms/pseries/vas.c | 86 +++++++++++++++++++++++ > arch/powerpc/platforms/pseries/vas.h | 6 ++ > 3 files changed, 97 insertions(+) > > diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c > index 85033f392c78..70004243e25e 100644 > --- a/arch/powerpc/platforms/pseries/mobility.c > +++ b/arch/powerpc/platforms/pseries/mobility.c > @@ -26,6 +26,7 @@ > #include <asm/machdep.h> > #include <asm/rtas.h> > #include "pseries.h" > +#include "vas.h" /* vas_migration_handler() */ > #include "../../kernel/cacheinfo.h" > > static struct kobject *mobility_kobj; > @@ -669,12 +670,16 @@ static int pseries_migrate_partition(u64 handle) > if (ret) > return ret; > > + vas_migration_handler(VAS_SUSPEND); > + vas_migration_handler() can return an error value. Is that OK to ignore before going into the suspend? > ret = pseries_suspend(handle); > if (ret == 0) > post_mobility_fixup(); > else > pseries_cancel_migration(handle, ret); > > + vas_migration_handler(VAS_RESUME); > + No concerns here though. Nothing to be done about errors encountered in the resume path. > return ret; > } > > diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c > index e4797fc73553..b53e3fe02971 100644 > --- a/arch/powerpc/platforms/pseries/vas.c > +++ b/arch/powerpc/platforms/pseries/vas.c > @@ -873,6 +873,92 @@ static struct notifier_block pseries_vas_nb = { > .notifier_call = pseries_vas_notifier, > }; > > +/* > + * For LPM, all windows have to be closed on the source partition > + * before migration and reopen them on the destination partition > + * after migration. So closing windows during suspend and > + * reopen them during resume. > + */ > +int vas_migration_handler(int action) > +{ > + struct hv_vas_cop_feat_caps *hv_caps; > + struct vas_cop_feat_caps *caps; > + int lpar_creds, new_creds = 0; > + struct vas_caps *vcaps; > + int i, rc = 0; > + > + hv_caps = kmalloc(sizeof(*hv_caps), GFP_KERNEL); > + if (!hv_caps) > + return -ENOMEM; > + > + mutex_lock(&vas_pseries_mutex); > + > + for (i = 0; i < VAS_MAX_FEAT_TYPE; i++) { > + vcaps = &vascaps[i]; > + caps = &vcaps->caps; > + lpar_creds = atomic_read(&caps->target_creds); > + > + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, > + vcaps->feat, > + (u64)virt_to_phys(hv_caps)); > + if (!rc) { > + new_creds = be16_to_cpu(hv_caps->target_lpar_creds); > + /* > + * Should not happen. But incase print messages, close > + * all windows in the list during suspend and reopen > + * windows based on new lpar_creds on the destination > + * system. > + */ > + if (lpar_creds != new_creds) { > + pr_err("state(%d): lpar creds: %d HV lpar creds: %d\n", > + action, lpar_creds, new_creds); > + pr_err("Used creds: %d, Active creds: %d\n", > + atomic_read(&caps->used_creds), > + vcaps->num_wins - vcaps->close_wins); > + } > + } else { > + pr_err("state(%d): Get VAS capabilities failed with %d\n", > + action, rc); > + /* > + * We can not stop migration with the current lpm > + * implementation. So continue closing all windows in > + * the list (during suspend) and return without > + * opening windows (during resume) if VAS capabilities > + * HCALL failed. > + */ > + if (action == VAS_RESUME) > + goto out; > + } > + > + switch (action) { > + case VAS_SUSPEND: > + rc = reconfig_close_windows(vcaps, vcaps->num_wins, > + true); > + break; > + case VAS_RESUME: > + atomic_set(&caps->target_creds, new_creds); > + rc = reconfig_open_windows(vcaps, new_creds, true); > + break; > + default: > + /* should not happen */ > + pr_err("Invalid migration action %d\n", action); > + rc = -EINVAL; > + goto out; > + } > + > + /* > + * Ignore errors during suspend and return for resume. > + */ > + if (rc && (action == VAS_RESUME)) > + goto out; > + } > + > +out: > + mutex_unlock(&vas_pseries_mutex); > + kfree(hv_caps); > + return rc; > +} The control flow with respect to releasing the allocation and the mutex looks correct to me. I also verified that H_QUERY_VAS_CAPABILITIES does not return a busy/retry status, so this code appears to handle all the architected statuses for that call.
On Mon, 2022-01-31 at 10:37 -0600, Nathan Lynch wrote: > Hi Haren, > > Mostly this seems OK to me. Some questions: Thanks Nathan for your suggestions. > > Haren Myneni <haren@linux.ibm.com> writes: > > Since the VAS windows belong to the VAS hardware resource, the > > hypervisor expects the partition to close them on source partition > > and reopen them after the partition migrated on the destination > > machine. > > Not clear to me what "expects" really means here. Would it be > accurate > to say "requires" instead? If the OS fails to close the windows > before > suspend, what happens? I will change it to 'requires' - These VAS windows have to be closed before migration so that these windows / credits will be available to other LPARs on the source machine. We should see failures only with HCALL to close windows. Since the migration can not be stopped, continue to close all windows (print error messages and ignore HCALL failures). These windows belong to VAS engine on source system, so can not be used them on the destination machine. > > > > This handler is called before pseries_suspend() to close these > > windows and again invoked after migration. All active windows > > for both default and QoS types will be closed and mark them > > in-active and reopened after migration with this handler. > > During the migration, the user space receives paste instruction > > failure if it issues copy/paste on these in-active windows. > > OK, I assume that's tolerable to existing users, is that correct? > I.e. > users should already be prepared to incur arbitrary paste instruction > failures. Yes, the user space or the nxz library has to fall back to SW solution (if the library supports) or manage with existing windows like closing the not frequently used ones. We expect the library to make the necessary modifications. > > > Signed-off-by: Haren Myneni <haren@linux.ibm.com> > > --- > > arch/powerpc/platforms/pseries/mobility.c | 5 ++ > > arch/powerpc/platforms/pseries/vas.c | 86 > > +++++++++++++++++++++++ > > arch/powerpc/platforms/pseries/vas.h | 6 ++ > > 3 files changed, 97 insertions(+) > > > > diff --git a/arch/powerpc/platforms/pseries/mobility.c > > b/arch/powerpc/platforms/pseries/mobility.c > > index 85033f392c78..70004243e25e 100644 > > --- a/arch/powerpc/platforms/pseries/mobility.c > > +++ b/arch/powerpc/platforms/pseries/mobility.c > > @@ -26,6 +26,7 @@ > > #include <asm/machdep.h> > > #include <asm/rtas.h> > > #include "pseries.h" > > +#include "vas.h" /* vas_migration_handler() */ > > #include "../../kernel/cacheinfo.h" > > > > static struct kobject *mobility_kobj; > > @@ -669,12 +670,16 @@ static int pseries_migrate_partition(u64 > > handle) > > if (ret) > > return ret; > > > > + vas_migration_handler(VAS_SUSPEND); > > + > > vas_migration_handler() can return an error value. Is that OK to > ignore > before going into the suspend? The migration continues even with these failures. So ignoring the failures from vas_migration_handler. Suspend operation: - we should expect failure with close HCALL or if the total credits does not match with the hypervisor (should not happen) - continue to close all windows even with HCALL failures with few windows since these HW resources are specific to machine. - Display error messages for HCALL failures (for debugging) Resume operation: - May see failures from HCALL to open/modify window HCALLs, but should not expect unless a bug in hypervisor or HW. - Stop opening windows if sees failure from HCALLs and print error messages - It does not stop migration, but the nxz library can see continue failures for many in-active windows. > > > ret = pseries_suspend(handle); > > if (ret == 0) > > post_mobility_fixup(); > > else > > pseries_cancel_migration(handle, ret); > > > > + vas_migration_handler(VAS_RESUME); > > + > > No concerns here though. Nothing to be done about errors encountered > in > the resume path. Correct as mentioned above ignoring error from VAS migration hanlder. Thanks Haren > > > return ret; > > } > > > > diff --git a/arch/powerpc/platforms/pseries/vas.c > > b/arch/powerpc/platforms/pseries/vas.c > > index e4797fc73553..b53e3fe02971 100644 > > --- a/arch/powerpc/platforms/pseries/vas.c > > +++ b/arch/powerpc/platforms/pseries/vas.c > > @@ -873,6 +873,92 @@ static struct notifier_block pseries_vas_nb = > > { > > .notifier_call = pseries_vas_notifier, > > }; > > > > +/* > > + * For LPM, all windows have to be closed on the source partition > > + * before migration and reopen them on the destination partition > > + * after migration. So closing windows during suspend and > > + * reopen them during resume. > > + */ > > +int vas_migration_handler(int action) > > +{ > > + struct hv_vas_cop_feat_caps *hv_caps; > > + struct vas_cop_feat_caps *caps; > > + int lpar_creds, new_creds = 0; > > + struct vas_caps *vcaps; > > + int i, rc = 0; > > + > > + hv_caps = kmalloc(sizeof(*hv_caps), GFP_KERNEL); > > + if (!hv_caps) > > + return -ENOMEM; > > + > > + mutex_lock(&vas_pseries_mutex); > > + > > + for (i = 0; i < VAS_MAX_FEAT_TYPE; i++) { > > + vcaps = &vascaps[i]; > > + caps = &vcaps->caps; > > + lpar_creds = atomic_read(&caps->target_creds); > > + > > + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, > > + vcaps->feat, > > + (u64)virt_to_phys(hv_caps > > )); > > + if (!rc) { > > + new_creds = be16_to_cpu(hv_caps- > > >target_lpar_creds); > > + /* > > + * Should not happen. But incase print > > messages, close > > + * all windows in the list during suspend and > > reopen > > + * windows based on new lpar_creds on the > > destination > > + * system. > > + */ > > + if (lpar_creds != new_creds) { > > + pr_err("state(%d): lpar creds: %d HV > > lpar creds: %d\n", > > + action, lpar_creds, new_creds); > > + pr_err("Used creds: %d, Active creds: > > %d\n", > > + atomic_read(&caps->used_creds), > > + vcaps->num_wins - vcaps- > > >close_wins); > > + } > > + } else { > > + pr_err("state(%d): Get VAS capabilities failed > > with %d\n", > > + action, rc); > > + /* > > + * We can not stop migration with the current > > lpm > > + * implementation. So continue closing all > > windows in > > + * the list (during suspend) and return without > > + * opening windows (during resume) if VAS > > capabilities > > + * HCALL failed. > > + */ > > + if (action == VAS_RESUME) > > + goto out; > > + } > > + > > + switch (action) { > > + case VAS_SUSPEND: > > + rc = reconfig_close_windows(vcaps, vcaps- > > >num_wins, > > + true); > > + break; > > + case VAS_RESUME: > > + atomic_set(&caps->target_creds, new_creds); > > + rc = reconfig_open_windows(vcaps, new_creds, > > true); > > + break; > > + default: > > + /* should not happen */ > > + pr_err("Invalid migration action %d\n", > > action); > > + rc = -EINVAL; > > + goto out; > > + } > > + > > + /* > > + * Ignore errors during suspend and return for resume. > > + */ > > + if (rc && (action == VAS_RESUME)) > > + goto out; > > + } > > + > > +out: > > + mutex_unlock(&vas_pseries_mutex); > > + kfree(hv_caps); > > + return rc; > > +} > > The control flow with respect to releasing the allocation and the > mutex > looks correct to me. I also verified that H_QUERY_VAS_CAPABILITIES > does > not return a busy/retry status, so this code appears to handle all > the > architected statuses for that call.
Haren Myneni <haren@linux.ibm.com> writes: > On Mon, 2022-01-31 at 10:37 -0600, Nathan Lynch wrote: >> Haren Myneni <haren@linux.ibm.com> writes: >> > Since the VAS windows belong to the VAS hardware resource, the >> > hypervisor expects the partition to close them on source partition >> > and reopen them after the partition migrated on the destination >> > machine. >> >> Not clear to me what "expects" really means here. Would it be >> accurate to say "requires" instead? If the OS fails to close the >> windows before suspend, what happens? > > I will change it to 'requires' - These VAS windows have to be closed > before migration so that these windows / credits will be available to > other LPARs on the source machine. I'll rephrase. I want to know whether the architecture (PAPR or whichever doc) imposes a requirement on the OS (i.e. using words like "must") to close the windows, or does it merely make a recommendation ("should"), or is it silent on the subject? In your testing, if Linux has windows open at the time of calling ibm,suspend-me, does the call fail or succeed?
diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c index 85033f392c78..70004243e25e 100644 --- a/arch/powerpc/platforms/pseries/mobility.c +++ b/arch/powerpc/platforms/pseries/mobility.c @@ -26,6 +26,7 @@ #include <asm/machdep.h> #include <asm/rtas.h> #include "pseries.h" +#include "vas.h" /* vas_migration_handler() */ #include "../../kernel/cacheinfo.h" static struct kobject *mobility_kobj; @@ -669,12 +670,16 @@ static int pseries_migrate_partition(u64 handle) if (ret) return ret; + vas_migration_handler(VAS_SUSPEND); + ret = pseries_suspend(handle); if (ret == 0) post_mobility_fixup(); else pseries_cancel_migration(handle, ret); + vas_migration_handler(VAS_RESUME); + return ret; } diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c index e4797fc73553..b53e3fe02971 100644 --- a/arch/powerpc/platforms/pseries/vas.c +++ b/arch/powerpc/platforms/pseries/vas.c @@ -873,6 +873,92 @@ static struct notifier_block pseries_vas_nb = { .notifier_call = pseries_vas_notifier, }; +/* + * For LPM, all windows have to be closed on the source partition + * before migration and reopen them on the destination partition + * after migration. So closing windows during suspend and + * reopen them during resume. + */ +int vas_migration_handler(int action) +{ + struct hv_vas_cop_feat_caps *hv_caps; + struct vas_cop_feat_caps *caps; + int lpar_creds, new_creds = 0; + struct vas_caps *vcaps; + int i, rc = 0; + + hv_caps = kmalloc(sizeof(*hv_caps), GFP_KERNEL); + if (!hv_caps) + return -ENOMEM; + + mutex_lock(&vas_pseries_mutex); + + for (i = 0; i < VAS_MAX_FEAT_TYPE; i++) { + vcaps = &vascaps[i]; + caps = &vcaps->caps; + lpar_creds = atomic_read(&caps->target_creds); + + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, + vcaps->feat, + (u64)virt_to_phys(hv_caps)); + if (!rc) { + new_creds = be16_to_cpu(hv_caps->target_lpar_creds); + /* + * Should not happen. But incase print messages, close + * all windows in the list during suspend and reopen + * windows based on new lpar_creds on the destination + * system. + */ + if (lpar_creds != new_creds) { + pr_err("state(%d): lpar creds: %d HV lpar creds: %d\n", + action, lpar_creds, new_creds); + pr_err("Used creds: %d, Active creds: %d\n", + atomic_read(&caps->used_creds), + vcaps->num_wins - vcaps->close_wins); + } + } else { + pr_err("state(%d): Get VAS capabilities failed with %d\n", + action, rc); + /* + * We can not stop migration with the current lpm + * implementation. So continue closing all windows in + * the list (during suspend) and return without + * opening windows (during resume) if VAS capabilities + * HCALL failed. + */ + if (action == VAS_RESUME) + goto out; + } + + switch (action) { + case VAS_SUSPEND: + rc = reconfig_close_windows(vcaps, vcaps->num_wins, + true); + break; + case VAS_RESUME: + atomic_set(&caps->target_creds, new_creds); + rc = reconfig_open_windows(vcaps, new_creds, true); + break; + default: + /* should not happen */ + pr_err("Invalid migration action %d\n", action); + rc = -EINVAL; + goto out; + } + + /* + * Ignore errors during suspend and return for resume. + */ + if (rc && (action == VAS_RESUME)) + goto out; + } + +out: + mutex_unlock(&vas_pseries_mutex); + kfree(hv_caps); + return rc; +} + static int __init pseries_vas_init(void) { struct hv_vas_cop_feat_caps *hv_cop_caps; diff --git a/arch/powerpc/platforms/pseries/vas.h b/arch/powerpc/platforms/pseries/vas.h index 4cf1d0ef66a5..36dd140af01b 100644 --- a/arch/powerpc/platforms/pseries/vas.h +++ b/arch/powerpc/platforms/pseries/vas.h @@ -27,6 +27,11 @@ #define VAS_GZIP_QOS_CAPABILITIES 0x56516F73477A6970 #define VAS_GZIP_DEFAULT_CAPABILITIES 0x56446566477A6970 +enum vas_migrate_action { + VAS_SUSPEND, + VAS_RESUME, +}; + /* * Co-processor feature - GZIP QoS windows or GZIP default windows */ @@ -127,4 +132,5 @@ struct pseries_vas_window { int sysfs_add_vas_caps(struct vas_cop_feat_caps *caps); int vas_reconfig_capabilties(u8 type); int __init sysfs_pseries_vas_init(struct vas_all_caps *vas_caps); +int vas_migration_handler(int action); #endif /* _VAS_H */
Since the VAS windows belong to the VAS hardware resource, the hypervisor expects the partition to close them on source partition and reopen them after the partition migrated on the destination machine. This handler is called before pseries_suspend() to close these windows and again invoked after migration. All active windows for both default and QoS types will be closed and mark them in-active and reopened after migration with this handler. During the migration, the user space receives paste instruction failure if it issues copy/paste on these in-active windows. Signed-off-by: Haren Myneni <haren@linux.ibm.com> --- arch/powerpc/platforms/pseries/mobility.c | 5 ++ arch/powerpc/platforms/pseries/vas.c | 86 +++++++++++++++++++++++ arch/powerpc/platforms/pseries/vas.h | 6 ++ 3 files changed, 97 insertions(+)