| Message ID | 168870305868.29142.5121604177475325995-4@git.sr.ht |
|---|---|
| State | New |
| Headers | show |
| Series | migration: introduce dirtylimit capability | expand |
~hyman <hyman@git.sr.ht> writes: > From: Hyman Huang(黄勇) <yong.huang@smartx.com> > > Introduce migration dirty-limit capability, which can > be turned on before live migration and limit dirty > page rate durty live migration. > > Introduce migrate_dirty_limit function to help check > if dirty-limit capability enabled during live migration. > > Meanwhile, refactor vcpu_dirty_rate_stat_collect > so that period can be configured instead of hardcoded. > > dirty-limit capability is kind of like auto-converge > but using dirty limit instead of traditional cpu-throttle > to throttle guest down. To enable this feature, turn on > the dirty-limit capability before live migration using > migrate-set-capabilities, and set the parameters > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably > to speed up convergence. > > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> > Acked-by: Peter Xu <peterx@redhat.com> > Reviewed-by: Juan Quintela <quintela@redhat.com> [...] > diff --git a/qapi/migration.json b/qapi/migration.json > index e43371955a..031832cde5 100644 > --- a/qapi/migration.json > +++ b/qapi/migration.json > @@ -497,6 +497,15 @@ > # are present. 'return-path' capability must be enabled to use > # it. (since 8.1) > # > +# @dirty-limit: If enabled, migration will use the dirty-limit > +# algorithm to throttle down guest instead of auto-converge > +# algorithm. This algorithm only works when vCPU's dirtyrate Two spaces after sentence-ending punctuation, please. "dirty rate" with a space, because that's how we spell it elsewhere. > +# greater than 'vcpu-dirty-limit', read processes in guest os > +# aren't penalized any more, so the algorithm can improve > +# performance of vCPU during live migration. This is an optional > +# performance feature and should not affect the correctness of the > +# existing auto-converge algorithm. (since 8.1) > +# I'm still confused. The text suggests there are two separate algorithms "to throttle down guest": "auto converge" and "dirty limit", and we get to pick one. Correct? If it is correct, then the last sentence feels redundant: picking another algorithm can't affect the algorithm we're *not* using. What are you trying to express here? When do we use "auto converge", and when do we use "dirty limit"? What does the user really need to know about these algorithms? Enough to pick one, I guess. That means advantages and disadvantages of the two algorithms. Which are? > # Features: > # > # @unstable: Members @x-colo and @x-ignore-shared are experimental. > @@ -512,7 +521,8 @@ > 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', > { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, > 'validate-uuid', 'background-snapshot', > - 'zero-copy-send', 'postcopy-preempt', 'switchover-ack'] } > + 'zero-copy-send', 'postcopy-preempt', 'switchover-ack', > + 'dirty-limit'] } > > ## > # @MigrationCapabilityStatus: [...]
On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> wrote: > ~hyman <hyman@git.sr.ht> writes: > > > From: Hyman Huang(黄勇) <yong.huang@smartx.com> > > > > Introduce migration dirty-limit capability, which can > > be turned on before live migration and limit dirty > > page rate durty live migration. > > > > Introduce migrate_dirty_limit function to help check > > if dirty-limit capability enabled during live migration. > > > > Meanwhile, refactor vcpu_dirty_rate_stat_collect > > so that period can be configured instead of hardcoded. > > > > dirty-limit capability is kind of like auto-converge > > but using dirty limit instead of traditional cpu-throttle > > to throttle guest down. To enable this feature, turn on > > the dirty-limit capability before live migration using > > migrate-set-capabilities, and set the parameters > > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably > > to speed up convergence. > > > > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> > > Acked-by: Peter Xu <peterx@redhat.com> > > Reviewed-by: Juan Quintela <quintela@redhat.com> > > [...] > > > diff --git a/qapi/migration.json b/qapi/migration.json > > index e43371955a..031832cde5 100644 > > --- a/qapi/migration.json > > +++ b/qapi/migration.json > > @@ -497,6 +497,15 @@ > > # are present. 'return-path' capability must be enabled to use > > # it. (since 8.1) > > # > > +# @dirty-limit: If enabled, migration will use the dirty-limit > > +# algorithm to throttle down guest instead of auto-converge > > +# algorithm. This algorithm only works when vCPU's dirtyrate > > Two spaces after sentence-ending punctuation, please. > > "dirty rate" with a space, because that's how we spell it elsewhere. > > > +# greater than 'vcpu-dirty-limit', read processes in guest os > > +# aren't penalized any more, so the algorithm can improve > > +# performance of vCPU during live migration. This is an optional > > +# performance feature and should not affect the correctness of the > > +# existing auto-converge algorithm. (since 8.1) > > +# > > I'm still confused. > > The text suggests there are two separate algorithms "to throttle down > guest": "auto converge" and "dirty limit", and we get to pick one. > Correct? > Yes, indeed ! > > If it is correct, then the last sentence feels redundant: picking > another algorithm can't affect the algorithm we're *not* using. What > are you trying to express here? > What i want to express is that the new algorithm implementation does not affect the original algorithm, leaving it in the comments seems redundant indeed. I'll drop this in the next version. > > When do we use "auto converge", and when do we use "dirty limit"? > > What does the user really need to know about these algorithms? Enough > to pick one, I guess. That means advantages and disadvantages of the > two algorithms. Which are? 1. The implementation of dirty-limit is based on dirty-ring, which is qualified to big systems with huge memories and can improve huge guest VM responsiveness remarkably during live migration. As a consequence, dirty-limit is recommended on platforms with huge guest VMs as is the way with dirty-ring. 2. dirty-limit convergence algorithm does not affect the "read-process" in guest VM, so guest VM gains the equal read performance nearly as it runs on host during the live migration. As a result, dirty-limit is recommended if the guest VM requires a stable read performance. The above explanation is about the recommendation of dirty-limit, please review, if it's ok, i'll place it in the comment of the dirty-limit capability. > > > # Features: > > # > > # @unstable: Members @x-colo and @x-ignore-shared are experimental. > > @@ -512,7 +521,8 @@ > > 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', > > { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, > > 'validate-uuid', 'background-snapshot', > > - 'zero-copy-send', 'postcopy-preempt', 'switchover-ack'] } > > + 'zero-copy-send', 'postcopy-preempt', 'switchover-ack', > > + 'dirty-limit'] } > > > > ## > > # @MigrationCapabilityStatus: > > [...] > > Thank Markus again for the attention to this patchset. :) Yong
Yong Huang <yong.huang@smartx.com> writes: > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> wrote: > >> ~hyman <hyman@git.sr.ht> writes: >> >> > From: Hyman Huang(黄勇) <yong.huang@smartx.com> >> > >> > Introduce migration dirty-limit capability, which can >> > be turned on before live migration and limit dirty >> > page rate durty live migration. >> > >> > Introduce migrate_dirty_limit function to help check >> > if dirty-limit capability enabled during live migration. >> > >> > Meanwhile, refactor vcpu_dirty_rate_stat_collect >> > so that period can be configured instead of hardcoded. >> > >> > dirty-limit capability is kind of like auto-converge >> > but using dirty limit instead of traditional cpu-throttle >> > to throttle guest down. To enable this feature, turn on >> > the dirty-limit capability before live migration using >> > migrate-set-capabilities, and set the parameters >> > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably >> > to speed up convergence. >> > >> > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> >> > Acked-by: Peter Xu <peterx@redhat.com> >> > Reviewed-by: Juan Quintela <quintela@redhat.com> >> >> [...] >> >> > diff --git a/qapi/migration.json b/qapi/migration.json >> > index e43371955a..031832cde5 100644 >> > --- a/qapi/migration.json >> > +++ b/qapi/migration.json >> > @@ -497,6 +497,15 @@ >> > # are present. 'return-path' capability must be enabled to use >> > # it. (since 8.1) >> > # >> > +# @dirty-limit: If enabled, migration will use the dirty-limit >> > +# algorithm to throttle down guest instead of auto-converge >> > +# algorithm. This algorithm only works when vCPU's dirtyrate >> >> Two spaces after sentence-ending punctuation, please. >> >> "dirty rate" with a space, because that's how we spell it elsewhere. >> >> > +# greater than 'vcpu-dirty-limit', read processes in guest os >> > +# aren't penalized any more, so the algorithm can improve >> > +# performance of vCPU during live migration. This is an optional >> > +# performance feature and should not affect the correctness of the >> > +# existing auto-converge algorithm. (since 8.1) >> > +# >> >> I'm still confused. >> >> The text suggests there are two separate algorithms "to throttle down >> guest": "auto converge" and "dirty limit", and we get to pick one. >> Correct? >> > Yes, indeed ! > >> >> If it is correct, then the last sentence feels redundant: picking >> another algorithm can't affect the algorithm we're *not* using. What >> are you trying to express here? >> > What i want to express is that the new algorithm implementation does > not affect the original algorithm, leaving it in the comments seems > redundant indeed. I'll drop this in the next version. Works for me. >> When do we use "auto converge", and when do we use "dirty limit"? >> >> What does the user really need to know about these algorithms? Enough >> to pick one, I guess. That means advantages and disadvantages of the >> two algorithms. Which are? > > 1. The implementation of dirty-limit is based on dirty-ring, which is > qualified > to big systems with huge memories and can improve huge guest VM > responsiveness remarkably during live migration. As a consequence, > dirty-limit > is recommended on platforms with huge guest VMs as is the way with > dirty-ring. > 2. dirty-limit convergence algorithm does not affect the "read-process" in > guest > VM, so guest VM gains the equal read performance nearly as it runs on > host > during the live migration. As a result, dirty-limit is recommended if > the guest > VM requires a stable read performance. > The above explanation is about the recommendation of dirty-limit, please > review, > if it's ok, i'll place it in the comment of the dirty-limit capability. Yes, please. But before that, I have still more questions. "This algorithm only works when vCPU's dirtyrate greater than 'vcpu-dirty-limit'" is a condition: "FEATURE only works when CONDITION". What happens when the condition is not met? How can the user ensure the condition is met? [...]
On Tue, Jul 18, 2023 at 7:04 PM Markus Armbruster <armbru@redhat.com> wrote: > Yong Huang <yong.huang@smartx.com> writes: > > > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> > wrote: > > > >> ~hyman <hyman@git.sr.ht> writes: > >> > >> > From: Hyman Huang(黄勇) <yong.huang@smartx.com> > >> > > >> > Introduce migration dirty-limit capability, which can > >> > be turned on before live migration and limit dirty > >> > page rate durty live migration. > >> > > >> > Introduce migrate_dirty_limit function to help check > >> > if dirty-limit capability enabled during live migration. > >> > > >> > Meanwhile, refactor vcpu_dirty_rate_stat_collect > >> > so that period can be configured instead of hardcoded. > >> > > >> > dirty-limit capability is kind of like auto-converge > >> > but using dirty limit instead of traditional cpu-throttle > >> > to throttle guest down. To enable this feature, turn on > >> > the dirty-limit capability before live migration using > >> > migrate-set-capabilities, and set the parameters > >> > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably > >> > to speed up convergence. > >> > > >> > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> > >> > Acked-by: Peter Xu <peterx@redhat.com> > >> > Reviewed-by: Juan Quintela <quintela@redhat.com> > >> > >> [...] > >> > >> > diff --git a/qapi/migration.json b/qapi/migration.json > >> > index e43371955a..031832cde5 100644 > >> > --- a/qapi/migration.json > >> > +++ b/qapi/migration.json > >> > @@ -497,6 +497,15 @@ > >> > # are present. 'return-path' capability must be enabled to use > >> > # it. (since 8.1) > >> > # > >> > +# @dirty-limit: If enabled, migration will use the dirty-limit > >> > +# algorithm to throttle down guest instead of auto-converge > >> > +# algorithm. This algorithm only works when vCPU's dirtyrate > >> > >> Two spaces after sentence-ending punctuation, please. > >> > >> "dirty rate" with a space, because that's how we spell it elsewhere. > >> > >> > +# greater than 'vcpu-dirty-limit', read processes in guest os > >> > +# aren't penalized any more, so the algorithm can improve > >> > +# performance of vCPU during live migration. This is an optional > >> > +# performance feature and should not affect the correctness of > the > >> > +# existing auto-converge algorithm. (since 8.1) > >> > +# > >> > >> I'm still confused. > >> > >> The text suggests there are two separate algorithms "to throttle down > >> guest": "auto converge" and "dirty limit", and we get to pick one. > >> Correct? > >> > > Yes, indeed ! > > > >> > >> If it is correct, then the last sentence feels redundant: picking > >> another algorithm can't affect the algorithm we're *not* using. What > >> are you trying to express here? > >> > > What i want to express is that the new algorithm implementation does > > not affect the original algorithm, leaving it in the comments seems > > redundant indeed. I'll drop this in the next version. > > Works for me. > > >> When do we use "auto converge", and when do we use "dirty limit"? > >> > >> What does the user really need to know about these algorithms? Enough > >> to pick one, I guess. That means advantages and disadvantages of the > >> two algorithms. Which are? > > > > 1. The implementation of dirty-limit is based on dirty-ring, which is > > qualified > > to big systems with huge memories and can improve huge guest VM > > responsiveness remarkably during live migration. As a consequence, > > dirty-limit > > is recommended on platforms with huge guest VMs as is the way with > > dirty-ring. > > 2. dirty-limit convergence algorithm does not affect the "read-process" > in > > guest > > VM, so guest VM gains the equal read performance nearly as it runs on > > host > > during the live migration. As a result, dirty-limit is recommended if > > the guest > > VM requires a stable read performance. > > The above explanation is about the recommendation of dirty-limit, please > > review, > > if it's ok, i'll place it in the comment of the dirty-limit capability. > > Yes, please. But before that, I have still more questions. "This > algorithm only works when vCPU's dirtyrate greater than > 'vcpu-dirty-limit'" is a condition: "FEATURE only works when CONDITION". > I failed to express my meaning again : ( . "Throttle algo only works when vCPU's dirtyrate greater than 'vcpu-dirty-limit' " should change to "vCPU throttle only works when vCPU's dirtyrate greater than 'vcpu-dirty-limit'". Not the whole "algo" ! > What happens when the condition is not met? How can the user ensure the > condition is met? > > [...] > >
Yong Huang <yong.huang@smartx.com> writes: > On Tue, Jul 18, 2023 at 7:04 PM Markus Armbruster <armbru@redhat.com> wrote: > >> Yong Huang <yong.huang@smartx.com> writes: >> >> > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> >> wrote: >> > >> >> ~hyman <hyman@git.sr.ht> writes: >> >> >> >> > From: Hyman Huang(黄勇) <yong.huang@smartx.com> >> >> > >> >> > Introduce migration dirty-limit capability, which can >> >> > be turned on before live migration and limit dirty >> >> > page rate durty live migration. >> >> > >> >> > Introduce migrate_dirty_limit function to help check >> >> > if dirty-limit capability enabled during live migration. >> >> > >> >> > Meanwhile, refactor vcpu_dirty_rate_stat_collect >> >> > so that period can be configured instead of hardcoded. >> >> > >> >> > dirty-limit capability is kind of like auto-converge >> >> > but using dirty limit instead of traditional cpu-throttle >> >> > to throttle guest down. To enable this feature, turn on >> >> > the dirty-limit capability before live migration using >> >> > migrate-set-capabilities, and set the parameters >> >> > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably >> >> > to speed up convergence. >> >> > >> >> > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> >> >> > Acked-by: Peter Xu <peterx@redhat.com> >> >> > Reviewed-by: Juan Quintela <quintela@redhat.com> >> >> >> >> [...] >> >> >> >> > diff --git a/qapi/migration.json b/qapi/migration.json >> >> > index e43371955a..031832cde5 100644 >> >> > --- a/qapi/migration.json >> >> > +++ b/qapi/migration.json >> >> > @@ -497,6 +497,15 @@ >> >> > # are present. 'return-path' capability must be enabled to use >> >> > # it. (since 8.1) >> >> > # >> >> > +# @dirty-limit: If enabled, migration will use the dirty-limit >> >> > +# algorithm to throttle down guest instead of auto-converge >> >> > +# algorithm. This algorithm only works when vCPU's dirtyrate >> >> >> >> Two spaces after sentence-ending punctuation, please. >> >> >> >> "dirty rate" with a space, because that's how we spell it elsewhere. >> >> >> >> > +# greater than 'vcpu-dirty-limit', read processes in guest os >> >> > +# aren't penalized any more, so the algorithm can improve >> >> > +# performance of vCPU during live migration. This is an optional >> >> > +# performance feature and should not affect the correctness of >> the >> >> > +# existing auto-converge algorithm. (since 8.1) >> >> > +# >> >> >> >> I'm still confused. >> >> >> >> The text suggests there are two separate algorithms "to throttle down >> >> guest": "auto converge" and "dirty limit", and we get to pick one. >> >> Correct? >> >> >> > Yes, indeed ! >> > >> >> >> >> If it is correct, then the last sentence feels redundant: picking >> >> another algorithm can't affect the algorithm we're *not* using. What >> >> are you trying to express here? >> >> >> > What i want to express is that the new algorithm implementation does >> > not affect the original algorithm, leaving it in the comments seems >> > redundant indeed. I'll drop this in the next version. >> >> Works for me. >> >> >> When do we use "auto converge", and when do we use "dirty limit"? >> >> >> >> What does the user really need to know about these algorithms? Enough >> >> to pick one, I guess. That means advantages and disadvantages of the >> >> two algorithms. Which are? >> > >> > 1. The implementation of dirty-limit is based on dirty-ring, which is >> > qualified >> > to big systems with huge memories and can improve huge guest VM >> > responsiveness remarkably during live migration. As a consequence, >> > dirty-limit >> > is recommended on platforms with huge guest VMs as is the way with >> > dirty-ring. >> > 2. dirty-limit convergence algorithm does not affect the "read-process" >> in >> > guest >> > VM, so guest VM gains the equal read performance nearly as it runs on >> > host >> > during the live migration. As a result, dirty-limit is recommended if >> > the guest >> > VM requires a stable read performance. >> > The above explanation is about the recommendation of dirty-limit, please >> > review, >> > if it's ok, i'll place it in the comment of the dirty-limit capability. >> >> Yes, please. But before that, I have still more questions. "This >> algorithm only works when vCPU's dirtyrate greater than >> 'vcpu-dirty-limit'" is a condition: "FEATURE only works when CONDITION". >> > I failed to express my meaning again : ( . "Throttle algo only works when > vCPU's dirtyrate greater than 'vcpu-dirty-limit' " should change to > "vCPU throttle only works when vCPU's dirtyrate greater than > 'vcpu-dirty-limit'". > Not the whole "algo" ! Let me paraphrase to make sure I got it... The vCPU is throttled as needed to keep its dirty rate within the limit set with set-vcpu-dirty-limit. Correct? What happens when I enable the dirty limit convergence algorithm without setting a limit with set-vcpu-dirty-limit? >> What happens when the condition is not met? How can the user ensure the >> condition is met? >> >> [...] >> >>
On Wed, Jul 19, 2023 at 1:26 PM Markus Armbruster <armbru@redhat.com> wrote: > Yong Huang <yong.huang@smartx.com> writes: > > > On Tue, Jul 18, 2023 at 7:04 PM Markus Armbruster <armbru@redhat.com> > wrote: > > > >> Yong Huang <yong.huang@smartx.com> writes: > >> > >> > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> > >> wrote: > >> > > >> >> ~hyman <hyman@git.sr.ht> writes: > >> >> > >> >> > From: Hyman Huang(黄勇) <yong.huang@smartx.com> > >> >> > > >> >> > Introduce migration dirty-limit capability, which can > >> >> > be turned on before live migration and limit dirty > >> >> > page rate durty live migration. > >> >> > > >> >> > Introduce migrate_dirty_limit function to help check > >> >> > if dirty-limit capability enabled during live migration. > >> >> > > >> >> > Meanwhile, refactor vcpu_dirty_rate_stat_collect > >> >> > so that period can be configured instead of hardcoded. > >> >> > > >> >> > dirty-limit capability is kind of like auto-converge > >> >> > but using dirty limit instead of traditional cpu-throttle > >> >> > to throttle guest down. To enable this feature, turn on > >> >> > the dirty-limit capability before live migration using > >> >> > migrate-set-capabilities, and set the parameters > >> >> > "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably > >> >> > to speed up convergence. > >> >> > > >> >> > Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> > >> >> > Acked-by: Peter Xu <peterx@redhat.com> > >> >> > Reviewed-by: Juan Quintela <quintela@redhat.com> > >> >> > >> >> [...] > >> >> > >> >> > diff --git a/qapi/migration.json b/qapi/migration.json > >> >> > index e43371955a..031832cde5 100644 > >> >> > --- a/qapi/migration.json > >> >> > +++ b/qapi/migration.json > >> >> > @@ -497,6 +497,15 @@ > >> >> > # are present. 'return-path' capability must be enabled to > use > >> >> > # it. (since 8.1) > >> >> > # > >> >> > +# @dirty-limit: If enabled, migration will use the dirty-limit > >> >> > +# algorithm to throttle down guest instead of auto-converge > >> >> > +# algorithm. This algorithm only works when vCPU's dirtyrate > >> >> > >> >> Two spaces after sentence-ending punctuation, please. > >> >> > >> >> "dirty rate" with a space, because that's how we spell it elsewhere. > >> >> > >> >> > +# greater than 'vcpu-dirty-limit', read processes in guest os > >> >> > +# aren't penalized any more, so the algorithm can improve > >> >> > +# performance of vCPU during live migration. This is an > optional > >> >> > +# performance feature and should not affect the correctness of > >> the > >> >> > +# existing auto-converge algorithm. (since 8.1) > >> >> > +# > >> >> > >> >> I'm still confused. > >> >> > >> >> The text suggests there are two separate algorithms "to throttle down > >> >> guest": "auto converge" and "dirty limit", and we get to pick one. > >> >> Correct? > >> >> > >> > Yes, indeed ! > >> > > >> >> > >> >> If it is correct, then the last sentence feels redundant: picking > >> >> another algorithm can't affect the algorithm we're *not* using. What > >> >> are you trying to express here? > >> >> > >> > What i want to express is that the new algorithm implementation does > >> > not affect the original algorithm, leaving it in the comments seems > >> > redundant indeed. I'll drop this in the next version. > >> > >> Works for me. > >> > >> >> When do we use "auto converge", and when do we use "dirty limit"? > >> >> > >> >> What does the user really need to know about these algorithms? > Enough > >> >> to pick one, I guess. That means advantages and disadvantages of the > >> >> two algorithms. Which are? > >> > > >> > 1. The implementation of dirty-limit is based on dirty-ring, which is > >> > qualified > >> > to big systems with huge memories and can improve huge guest VM > >> > responsiveness remarkably during live migration. As a consequence, > >> > dirty-limit > >> > is recommended on platforms with huge guest VMs as is the way with > >> > dirty-ring. > >> > 2. dirty-limit convergence algorithm does not affect the > "read-process" > >> in > >> > guest > >> > VM, so guest VM gains the equal read performance nearly as it runs > on > >> > host > >> > during the live migration. As a result, dirty-limit is recommended > if > >> > the guest > >> > VM requires a stable read performance. > >> > The above explanation is about the recommendation of dirty-limit, > please > >> > review, > >> > if it's ok, i'll place it in the comment of the dirty-limit > capability. > >> > >> Yes, please. But before that, I have still more questions. "This > >> algorithm only works when vCPU's dirtyrate greater than > >> 'vcpu-dirty-limit'" is a condition: "FEATURE only works when CONDITION". > >> > > I failed to express my meaning again : ( . "Throttle algo only works > when > > vCPU's dirtyrate greater than 'vcpu-dirty-limit' " should change to > > "vCPU throttle only works when vCPU's dirtyrate greater than > > 'vcpu-dirty-limit'". > > Not the whole "algo" ! > > Let me paraphrase to make sure I got it... The vCPU is throttled as > needed to keep its dirty rate within the limit set with > set-vcpu-dirty-limit. Correct? > Yes. Actually set with the internal function qmp_set_vcpu_dirty_limit. And a parameter called "vcpu-dirty-limit" of migration provided by dirty-limit aims to be the argument of qmp_set_vcpu_dirty_limit. > What happens when I enable the dirty limit convergence algorithm without > setting a limit with set-vcpu-dirty-limit? > dirty-limit will use the default value which is defined in migration/options.c: #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ So the default of the dirty-limit is 1MB/s. > > >> What happens when the condition is not met? How can the user ensure the > >> condition is met? > >> > >> [...] > >> > >> > >
Yong Huang <yong.huang@smartx.com> writes: > On Wed, Jul 19, 2023 at 1:26 PM Markus Armbruster <armbru@redhat.com> wrote: > >> Yong Huang <yong.huang@smartx.com> writes: >> >> > On Tue, Jul 18, 2023 at 7:04 PM Markus Armbruster <armbru@redhat.com> >> wrote: >> > >> >> Yong Huang <yong.huang@smartx.com> writes: >> >> >> >> > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster <armbru@redhat.com> wrote: [...] >> >> Yes, please. But before that, I have still more questions. "This >> >> algorithm only works when vCPU's dirtyrate greater than >> >> 'vcpu-dirty-limit'" is a condition: "FEATURE only works when CONDITION". >> >> >> > I failed to express my meaning again : ( . "Throttle algo only works when >> > vCPU's dirtyrate greater than 'vcpu-dirty-limit' " should change to >> > "vCPU throttle only works when vCPU's dirtyrate greater than >> > 'vcpu-dirty-limit'". >> > Not the whole "algo" ! >> >> Let me paraphrase to make sure I got it... The vCPU is throttled as >> needed to keep its dirty rate within the limit set with >> set-vcpu-dirty-limit. Correct? >> > Yes. Actually set with the internal function qmp_set_vcpu_dirty_limit. > > And a parameter called "vcpu-dirty-limit" of migration provided by > dirty-limit > aims to be the argument of qmp_set_vcpu_dirty_limit. Alright, let me try to craft some documentation: # @dirty-limit: If enabled, migration will throttle vCPUs as needed to # keep their dirty page rate within @vcpu-dirty-limit. This can # improve responsiveness of large guests during live migration, # and can result in more stable read performance. Requires KVM # with accelerator property "dirty-ring-size" set. (Since 8.1) What do you think? >> What happens when I enable the dirty limit convergence algorithm without >> setting a limit with set-vcpu-dirty-limit? >> > dirty-limit will use the default value which is defined > in migration/options.c: > #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ > > So the default of the dirty-limit is 1MB/s. Is this default documented in the QAPI schema? Hmm, looks like it isn't before this series, but PATCH 3 fixes it. Okay. >> >> What happens when the condition is not met? How can the user ensure the >> >> condition is met? >> >> >> >> [...]
On Wed, Jul 19, 2023 at 5:03 PM Markus Armbruster <armbru@redhat.com> wrote: > Yong Huang <yong.huang@smartx.com> writes: > > > On Wed, Jul 19, 2023 at 1:26 PM Markus Armbruster <armbru@redhat.com> > wrote: > > > >> Yong Huang <yong.huang@smartx.com> writes: > >> > >> > On Tue, Jul 18, 2023 at 7:04 PM Markus Armbruster <armbru@redhat.com> > >> wrote: > >> > > >> >> Yong Huang <yong.huang@smartx.com> writes: > >> >> > >> >> > On Thu, Jul 13, 2023 at 8:44 PM Markus Armbruster < > armbru@redhat.com> wrote: > > [...] > > >> >> Yes, please. But before that, I have still more questions. "This > >> >> algorithm only works when vCPU's dirtyrate greater than > >> >> 'vcpu-dirty-limit'" is a condition: "FEATURE only works when > CONDITION". > >> >> > >> > I failed to express my meaning again : ( . "Throttle algo only works > when > >> > vCPU's dirtyrate greater than 'vcpu-dirty-limit' " should change to > >> > "vCPU throttle only works when vCPU's dirtyrate greater than > >> > 'vcpu-dirty-limit'". > >> > Not the whole "algo" ! > >> > >> Let me paraphrase to make sure I got it... The vCPU is throttled as > >> needed to keep its dirty rate within the limit set with > >> set-vcpu-dirty-limit. Correct? > >> > > Yes. Actually set with the internal function qmp_set_vcpu_dirty_limit. > > > > And a parameter called "vcpu-dirty-limit" of migration provided by > > dirty-limit > > aims to be the argument of qmp_set_vcpu_dirty_limit. > > Alright, let me try to craft some documentation: > > # @dirty-limit: If enabled, migration will throttle vCPUs as needed to > # keep their dirty page rate within @vcpu-dirty-limit. This can > # improve responsiveness of large guests during live migration, > # and can result in more stable read performance. Requires KVM > # with accelerator property "dirty-ring-size" set. (Since 8.1) > > What do you think? > I feel ok, it seems clear and concise. I'll use this comment in the next version. Thanks a lot. :) > > >> What happens when I enable the dirty limit convergence algorithm without > >> setting a limit with set-vcpu-dirty-limit? > >> > > dirty-limit will use the default value which is defined > > in migration/options.c: > > #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ > > > > So the default of the dirty-limit is 1MB/s. > > Is this default documented in the QAPI schema? Hmm, looks like it isn't > before this series, but PATCH 3 fixes it. Okay. > > >> >> What happens when the condition is not met? How can the user ensure > the > >> >> condition is met? > >> >> > >> >> [...] > >
diff --git a/migration/options.c b/migration/options.c index 7d2d98830e..631c12cf32 100644 --- a/migration/options.c +++ b/migration/options.c @@ -27,6 +27,7 @@ #include "qemu-file.h" #include "ram.h" #include "options.h" +#include "sysemu/kvm.h" /* Maximum migrate downtime set to 2000 seconds */ #define MAX_MIGRATE_DOWNTIME_SECONDS 2000 @@ -196,6 +197,8 @@ Property migration_properties[] = { #endif DEFINE_PROP_MIG_CAP("x-switchover-ack", MIGRATION_CAPABILITY_SWITCHOVER_ACK), + DEFINE_PROP_MIG_CAP("x-dirty-limit", + MIGRATION_CAPABILITY_DIRTY_LIMIT), DEFINE_PROP_END_OF_LIST(), }; @@ -242,6 +245,13 @@ bool migrate_dirty_bitmaps(void) return s->capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS]; } +bool migrate_dirty_limit(void) +{ + MigrationState *s = migrate_get_current(); + + return s->capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT]; +} + bool migrate_events(void) { MigrationState *s = migrate_get_current(); @@ -573,6 +583,20 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } } + if (new_caps[MIGRATION_CAPABILITY_DIRTY_LIMIT]) { + if (new_caps[MIGRATION_CAPABILITY_AUTO_CONVERGE]) { + error_setg(errp, "dirty-limit conflicts with auto-converge" + " either of then available currently"); + return false; + } + + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty-limit requires KVM with accelerator" + " property 'dirty-ring-size' set"); + return false; + } + } + return true; } diff --git a/migration/options.h b/migration/options.h index 9aaf363322..b5a950d4e4 100644 --- a/migration/options.h +++ b/migration/options.h @@ -24,6 +24,7 @@ extern Property migration_properties[]; /* capabilities */ bool migrate_auto_converge(void); +bool migrate_dirty_limit(void); bool migrate_background_snapshot(void); bool migrate_block(void); bool migrate_colo(void); diff --git a/qapi/migration.json b/qapi/migration.json index e43371955a..031832cde5 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -497,6 +497,15 @@ # are present. 'return-path' capability must be enabled to use # it. (since 8.1) # +# @dirty-limit: If enabled, migration will use the dirty-limit +# algorithm to throttle down guest instead of auto-converge +# algorithm. This algorithm only works when vCPU's dirtyrate +# greater than 'vcpu-dirty-limit', read processes in guest os +# aren't penalized any more, so the algorithm can improve +# performance of vCPU during live migration. This is an optional +# performance feature and should not affect the correctness of the +# existing auto-converge algorithm. (since 8.1) +# # Features: # # @unstable: Members @x-colo and @x-ignore-shared are experimental. @@ -512,7 +521,8 @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt', 'switchover-ack'] } + 'zero-copy-send', 'postcopy-preempt', 'switchover-ack', + 'dirty-limit'] } ## # @MigrationCapabilityStatus: diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 5c12d26d49..953ef934bc 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -24,6 +24,9 @@ #include "hw/boards.h" #include "sysemu/kvm.h" #include "trace.h" +#include "migration/misc.h" +#include "migration/migration.h" +#include "migration/options.h" /* * Dirtylimit stop working if dirty page rate error @@ -75,11 +78,18 @@ static bool dirtylimit_quit; static void vcpu_dirty_rate_stat_collect(void) { + MigrationState *s = migrate_get_current(); VcpuStat stat; int i = 0; + int64_t period = DIRTYLIMIT_CALC_TIME_MS; + + if (migrate_dirty_limit() && + migration_is_active(s)) { + period = s->parameters.x_vcpu_dirty_limit_period; + } /* calculate vcpu dirtyrate */ - vcpu_calculate_dirtyrate(DIRTYLIMIT_CALC_TIME_MS, + vcpu_calculate_dirtyrate(period, &stat, GLOBAL_DIRTY_LIMIT, false);