| Message ID | 20220614082912.378355-1-fbarrat@linux.ibm.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | target/ppc: cpu_init: Clean up stop state on cpu reset | expand |
Frederic Barrat <fbarrat@linux.ibm.com> writes: > The 'resume_as_sreset' attribute of a cpu can be set when a thread is > entering a stop state on ppc books. It causes the thread to be > re-routed to vector 0x100 when woken up by an exception. So it must be > cleaned on reset or a thread might be re-routed unexpectedly after a > reset, when it was not in a stop state and/or when the appropriate > exception handler isn't set up yet. > > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> Reviewed-by: Fabiano Rosas <farosas@linux.ibm.com>
On 6/14/22 10:29, Frederic Barrat wrote: > The 'resume_as_sreset' attribute of a cpu can be set when a thread is > entering a stop state on ppc books. It causes the thread to be > re-routed to vector 0x100 when woken up by an exception. So it must be > cleaned on reset or a thread might be re-routed unexpectedly after a > reset, when it was not in a stop state and/or when the appropriate > exception handler isn't set up yet. What is the test scenario ? and what are the symptoms ? > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> Reviewed-by: Cédric Le Goater <clg@kaod.org> > --- > > I didn't find an appropriate commit to add a "Fixes:". It originates > when adding support for power management states but the code looked > quite different in 2016 and it's not clear whether we were supporting > reset then. It was added when we needed some support for the POWER8 stop states. About that time. Thanks, C. > > target/ppc/cpu_init.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c > index 0f891afa04..c16cb8dbe7 100644 > --- a/target/ppc/cpu_init.c > +++ b/target/ppc/cpu_init.c > @@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev) > } > pmu_update_summaries(env); > } > + > + /* clean any pending stop state */ > + env->resume_as_sreset = 0; > #endif > hreg_compute_hflags(env); > env->reserve_addr = (target_ulong)-1ULL;
On 15/06/2022 07:23, Cédric Le Goater wrote: > On 6/14/22 10:29, Frederic Barrat wrote: >> The 'resume_as_sreset' attribute of a cpu can be set when a thread is >> entering a stop state on ppc books. It causes the thread to be >> re-routed to vector 0x100 when woken up by an exception. So it must be >> cleaned on reset or a thread might be re-routed unexpectedly after a >> reset, when it was not in a stop state and/or when the appropriate >> exception handler isn't set up yet. > > What is the test scenario ? and what are the symptoms ? I was hitting it because of another bug in skiboot: if you have many chips, we spend way too much time in add_opal_interrupts(), especially on powernv10 (I'm working on a separate patch in skiboot to fix that). Sufficiently so that the watchdog timer resets the system. When it happens, all the secondary threads are in stopped state, only the main thread is working. That's how I was reproducing. What happens after the reset can vary a bit due to timing, but the most likely scenario is that we go through another primary thread election in skiboot. If the primary thread is the same as before, then there's no problem. If it's a different primary, then it will enter main_cpu_entry() while the other threads wait as secondaries. At some point, the primary thread (which still carries the wrong resume_as_sreset value from before reset) will enable the decrementer interrupt. The vector for the decrementer exception 0x900 is defined, so that shouldn't be a problem. However, because of the wrong resume_as_sreset value, it is re-routed to vector 0x100, which is still defined as the default boot-time handler, which is the entry point for BML. So the thread restarts as new, but this time it will be elected secondary. And we end up with all threads waiting as secondaries and a system stuck. All that happen before we've init the uart, so there's not a single trace on the console. Fun :-) Fred > > >> Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> > > Reviewed-by: Cédric Le Goater <clg@kaod.org> > > >> --- >> >> I didn't find an appropriate commit to add a "Fixes:". It originates >> when adding support for power management states but the code looked >> quite different in 2016 and it's not clear whether we were supporting >> reset then. > > It was added when we needed some support for the POWER8 stop states. > About that time. > > Thanks, > > C. > >> >> target/ppc/cpu_init.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c >> index 0f891afa04..c16cb8dbe7 100644 >> --- a/target/ppc/cpu_init.c >> +++ b/target/ppc/cpu_init.c >> @@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev) >> } >> pmu_update_summaries(env); >> } >> + >> + /* clean any pending stop state */ >> + env->resume_as_sreset = 0; >> #endif >> hreg_compute_hflags(env); >> env->reserve_addr = (target_ulong)-1ULL; > >
On 6/15/22 09:17, Frederic Barrat wrote: > > > On 15/06/2022 07:23, Cédric Le Goater wrote: >> On 6/14/22 10:29, Frederic Barrat wrote: >>> The 'resume_as_sreset' attribute of a cpu can be set when a thread is >>> entering a stop state on ppc books. It causes the thread to be >>> re-routed to vector 0x100 when woken up by an exception. So it must be >>> cleaned on reset or a thread might be re-routed unexpectedly after a >>> reset, when it was not in a stop state and/or when the appropriate >>> exception handler isn't set up yet. >> >> What is the test scenario ? and what are the symptoms ? > > > I was hitting it because of another bug in skiboot: if you have many chips, we spend way too much time in add_opal_interrupts(), especially on powernv10 (I'm working on a separate patch in skiboot to fix that). Sufficiently so that the watchdog timer resets the system. When it happens, all the secondary threads are in stopped state, only the main thread is working. That's how I was reproducing. > > What happens after the reset can vary a bit due to timing, but the most likely scenario is that we go through another primary thread election in skiboot. If the primary thread is the same as before, then there's no problem. If it's a different primary, then it will enter main_cpu_entry() while the other threads wait as secondaries. At some point, the primary thread (which still carries the wrong resume_as_sreset value from before reset) will enable the decrementer interrupt. The vector for the decrementer exception 0x900 is defined, so that shouldn't be a problem. However, because of the wrong resume_as_sreset value, it is re-routed to vector 0x100, which is still defined as the default boot-time handler, which is the entry point for BML. So the thread restarts as new, but this time it will be elected secondary. And we end up with all threads waiting as secondaries and a system stuck. All that happen before we've init the uart, so there's not a single trace on the console. > Fun :-) Great analysis ! I think this deserve a v2 just to put in the commit log what you just wrote :) Thanks, C.
diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c index 0f891afa04..c16cb8dbe7 100644 --- a/target/ppc/cpu_init.c +++ b/target/ppc/cpu_init.c @@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev) } pmu_update_summaries(env); } + + /* clean any pending stop state */ + env->resume_as_sreset = 0; #endif hreg_compute_hflags(env); env->reserve_addr = (target_ulong)-1ULL;
The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet. Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> --- I didn't find an appropriate commit to add a "Fixes:". It originates when adding support for power management states but the code looked quite different in 2016 and it's not clear whether we were supporting reset then. target/ppc/cpu_init.c | 3 +++ 1 file changed, 3 insertions(+)