Message ID | 1421301930-10035-1-git-send-email-dongsheng.wang@freescale.com (mailing list archive) |
---|---|
State | Changes Requested |
Delegated to: | Scott Wood |
Headers | show |
Hi all, U-boot patch link: http://patchwork.ozlabs.org/patch/429265/ Regards, -Dongsheng > -----Original Message----- > From: Dongsheng Wang [mailto:dongsheng.wang@freescale.com] > Sent: Thursday, January 15, 2015 2:06 PM > To: Wood Scott-B07421; Sun York-R58495; Li Yang-Leo-R58472 > Cc: linuxppc-dev@lists.ozlabs.org; Wang Dongsheng-B40534 > Subject: [PATCH] fsl/smp: add low power boot support to replace spin boot > > From: Wang Dongsheng <dongsheng.wang@freescale.com> > > U-boot put non-boot cpus into an low power state(PW10/PW20 or DOZE) when cpu > powered up. To exit low power state kernel will send DOORBELL or MPIC-IPI > signal to all those CPUs. > > e500/e500v2 use mpic to send IPI signal. > e500mc and later use doorbell to send IPI signal. > > This feature tested on: > POWER UP TEST: > P1022DS(e500v2),96k times. > P4080(e500mc), 110k times. > T1024(e5500), 83k times. > T4240(e6500), 150k times. > > CPU HOTPLUG TEST: > P1022DS(e500v2),1.4 million times. > P4080(e500mc), 1.8 million times. > T1024(e5500), 1.3 million times. > T4240(e6500), 1.1 million times. > > Signed-off-by: Wang Dongsheng <dongsheng.wang@freescale.com> > > diff --git a/arch/powerpc/include/asm/mpic.h b/arch/powerpc/include/asm/mpic.h > index 754f93d..8af6a25 100644 > --- a/arch/powerpc/include/asm/mpic.h > +++ b/arch/powerpc/include/asm/mpic.h > @@ -474,6 +474,15 @@ extern int mpic_cpu_get_priority(void); > /* Set the current cpu priority for this cpu */ > extern void mpic_cpu_set_priority(int prio); > > +/* Set cpu priority */ > +void mpic_set_cpu_priority(int nr, int prio); > + > +/* Set cpu EOI */ > +void mpic_cpu_eoi_write(int cpu); > + > +/* CPU ACK interrupt */ > +void mpic_cpu_ack(int cpu); > + > /* Request IPIs on primary mpic */ > extern void mpic_request_ipis(void); > > diff --git a/arch/powerpc/platforms/85xx/smp.c > b/arch/powerpc/platforms/85xx/smp.c > index d7c1e69..6c54632 100644 > --- a/arch/powerpc/platforms/85xx/smp.c > +++ b/arch/powerpc/platforms/85xx/smp.c > @@ -193,6 +193,30 @@ static int smp_85xx_kick_cpu(int nr) > const u64 *cpu_rel_addr; > __iomem struct epapr_spin_table *spin_table; > struct device_node *np; > + > + /* > + * DOORBELL: > + * When kernel kick one of cpus, all cpus will be wakenup. To make > + * sure that only the target cpu is effected, other cpus (by checking > + * spin_table->addr_l) should go back to low power state. > + * > + * U-boot has renumber the cpu PIR Why we need to set all of PIR to > + * the same value? > + * A: Before kernel kicking cpu, the doorbell message was not configured > + * for target cpu(cpu_messages->data). If we try to send a > + * non-configured message to target cpu, it cannot correctly receive > + * doorbell interrput. So SET ALL OF CPU'S PIR to the same value to > + * let all cpus catch the interrupt. > + * > + * Why set PIR to zero? > + * A: U-boot cannot know how many cpus will be kicked up(Kernel allow us > + * to configure NR_CPUS) and IPI is a per_cpu variable, u-boot cannot > + * set a appropriate PIR for every cpu, but the boot cpu(CPU0) always be > + * there. U-boot set PIR to zero as a default PIR ID for each CPU, so > + * initialize the kick_cpus to 0. > + */ > + u32 kick_cpus = 0; > + > int hw_cpu = get_hard_smp_processor_id(nr); > int ioremappable; > int ret = 0; > @@ -251,8 +275,7 @@ static int smp_85xx_kick_cpu(int nr) > spin_table = phys_to_virt(*cpu_rel_addr); > > local_irq_save(flags); > -#ifdef CONFIG_PPC32 > -#ifdef CONFIG_HOTPLUG_CPU > +#if defined(CONFIG_PPC32) && defined(CONFIG_HOTPLUG_CPU) > /* Corresponding to generic_set_cpu_dead() */ > generic_set_cpu_up(nr); > > @@ -292,11 +315,58 @@ static int smp_85xx_kick_cpu(int nr) > __secondary_hold_acknowledge = -1; > } > #endif > + > flush_spin_table(spin_table); > - out_be32(&spin_table->pir, hw_cpu); > + /* > + * U-boot will wait kernel send eoi to MPIC, after EOI has send > + * kernel will set PIR for uboot, let uboot know EOI has send. > + */ > + out_be32(&spin_table->pir, 0); > + > +#ifdef CONFIG_PPC32 > out_be32(&spin_table->addr_l, __pa(__early_start)); > +#else > + out_be64((u64 *)(&spin_table->addr_h), > + __pa(ppc_function_entry(generic_secondary_smp_init))); > +#endif > flush_spin_table(spin_table); > > + /* > + * e500, e500v2 need to use MPIC to send IPI signal, so we need to > + * open IPI firstly. > + */ > + if (!cpu_has_feature(CPU_FTR_DBELL)) { > + mpic_set_cpu_priority(nr, 0); > + kick_cpus = nr; > + } > + > + /* Let cpu exit low power state, and from u-boot jump to kernel */ > + arch_send_call_function_single_ipi(kick_cpus); > + > + /* > + * Let we ACK interrput and Send EOI signal to finish INT server > + * U-boot has read EPR to ACK interrput when MPIC work in external > + * proxy mode. Without the external proxy facility, we need to read > + * MPIC ACK register. > + * > + * There just ACK interrput, we don't need to get the interrupt vector > + * and to handle it. Because there just IPI or DOORBELL interrupt to > + * make u-boot exit low power state and jump to kernel. > + */ > + mpic_cpu_ack(nr); > + /* Send EOI to clear ISR bit to remove interrupt from service */ > + mpic_cpu_eoi_write(nr); > + > + /* After wakeup CPU disable IPI, IPI will be opened in setup_cpu */ > + if (!cpu_has_feature(CPU_FTR_DBELL)) > + mpic_set_cpu_priority(nr, 0xf); > + > + /* After EOI finish, let we release cpu */ > + flush_spin_table(spin_table); > + out_be32(&spin_table->pir, hw_cpu); > + flush_spin_table(spin_table); > + > +#ifdef CONFIG_PPC32 > /* Wait a bit for the CPU to ack. */ > if (!spin_event_timeout(__secondary_hold_acknowledge == hw_cpu, > 10000, 100)) { > @@ -308,12 +378,6 @@ static int smp_85xx_kick_cpu(int nr) > out: > #else > smp_generic_kick_cpu(nr); > - > - flush_spin_table(spin_table); > - out_be32(&spin_table->pir, hw_cpu); > - out_be64((u64 *)(&spin_table->addr_h), > - __pa(ppc_function_entry(generic_secondary_smp_init))); > - flush_spin_table(spin_table); > #endif > > local_irq_restore(flags); > diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c > index c4648ad..b2ba47e 100644 > --- a/arch/powerpc/sysdev/mpic.c > +++ b/arch/powerpc/sysdev/mpic.c > @@ -658,6 +658,21 @@ static inline void mpic_eoi(struct mpic *mpic) > (void)mpic_cpu_read(MPIC_INFO(CPU_WHOAMI)); > } > > +void mpic_cpu_eoi_write(int cpu) > +{ > + struct mpic *mpic = mpic_primary; > + > + _mpic_write(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_EOI), 0); > + _mpic_read(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_WHOAMI)); > +} > + > +void mpic_cpu_ack(int cpu) > +{ > + struct mpic *mpic = mpic_primary; > + > + _mpic_read(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_INTACK)); > +} > + > /* > * Linux descriptor level callbacks > */ > @@ -1778,6 +1793,16 @@ void mpic_cpu_set_priority(int prio) > mpic_cpu_write(MPIC_INFO(CPU_CURRENT_TASK_PRI), prio); > } > > +void mpic_set_cpu_priority(int nr, int prio) > +{ > + struct mpic *mpic = mpic_primary; > + int hw_cpu = get_hard_smp_processor_id(nr); > + > + prio &= MPIC_CPU_TASKPRI_MASK; > + _mpic_write(mpic->reg_type, &mpic->cpuregs[hw_cpu], > + MPIC_INFO(CPU_CURRENT_TASK_PRI), prio); > +} > + > void mpic_teardown_this_cpu(int secondary) > { > struct mpic *mpic = mpic_primary; > -- > 2.1.0.27.g96db324
On 01/14/2015 10:05 PM, Dongsheng Wang wrote: > From: Wang Dongsheng <dongsheng.wang@freescale.com> > > U-boot put non-boot cpus into an low power state(PW10/PW20 or DOZE) when cpu > powered up. To exit low power state kernel will send DOORBELL or MPIC-IPI > signal to all those CPUs. > > e500/e500v2 use mpic to send IPI signal. > e500mc and later use doorbell to send IPI signal. > > This feature tested on: > POWER UP TEST: > P1022DS(e500v2),96k times. > P4080(e500mc), 110k times. > T1024(e5500), 83k times. > T4240(e6500), 150k times. > > CPU HOTPLUG TEST: > P1022DS(e500v2),1.4 million times. > P4080(e500mc), 1.8 million times. > T1024(e5500), 1.3 million times. > T4240(e6500), 1.1 million times. > > Signed-off-by: Wang Dongsheng <dongsheng.wang@freescale.com> > This is a very good move. Can you explain what has been tested for 100K times, or over a million times? Have you verified on older e500v2 platforms, such as MPC8572? York
Thanks for your review. > -----Original Message----- > From: Sun York-R58495 > Sent: Friday, January 16, 2015 1:09 AM > To: Wang Dongsheng-B40534; Wood Scott-B07421; Li Yang-Leo-R58472 > Cc: linuxppc-dev@lists.ozlabs.org > Subject: Re: [PATCH] fsl/smp: add low power boot support to replace spin boot > > > > On 01/14/2015 10:05 PM, Dongsheng Wang wrote: > > From: Wang Dongsheng <dongsheng.wang@freescale.com> > > > > U-boot put non-boot cpus into an low power state(PW10/PW20 or DOZE) > > when cpu powered up. To exit low power state kernel will send DOORBELL > > or MPIC-IPI signal to all those CPUs. > > > > e500/e500v2 use mpic to send IPI signal. > > e500mc and later use doorbell to send IPI signal. > > > > This feature tested on: > > POWER UP TEST: > > P1022DS(e500v2),96k times. > > P4080(e500mc), 110k times. > > T1024(e5500), 83k times. > > T4240(e6500), 150k times. > > > > CPU HOTPLUG TEST: > > P1022DS(e500v2),1.4 million times. > > P4080(e500mc), 1.8 million times. > > T1024(e5500), 1.3 million times. > > T4240(e6500), 1.1 million times. > > > > Signed-off-by: Wang Dongsheng <dongsheng.wang@freescale.com> > > > > This is a very good move. Can you explain what has been tested for 100K times, > or over a million times? > Thanks, I tested linux boot process and cpu hotplug. For Linux boot process test: Each CPU can normally be started and random run an application, the application works well. For cpu hotplug test: Just plug out and plug in for each cpu. Cpu hotplug test works well. If I missed some case please let me know. > Have you verified on older e500v2 platforms, such as MPC8572? > I haven't tested MPC8572 and Just test P1022DS. Because is e500 and e500v2 use the same Low power boot codes. I will test this feature on MPC8572 platform. Regards, -Dongsheng
On Thu, 2015-01-15 at 14:05 +0800, Dongsheng Wang wrote: > From: Wang Dongsheng <dongsheng.wang@freescale.com> > > U-boot put non-boot cpus into an low power state(PW10/PW20 or DOZE) when cpu > powered up. To exit low power state kernel will send DOORBELL or MPIC-IPI > signal to all those CPUs. U-Boot does not do this yet. A patch was just posted to propose such a change. > @@ -292,11 +315,58 @@ static int smp_85xx_kick_cpu(int nr) > __secondary_hold_acknowledge = -1; > } > #endif > + > flush_spin_table(spin_table); > - out_be32(&spin_table->pir, hw_cpu); > + /* > + * U-boot will wait kernel send eoi to MPIC, after EOI has send > + * kernel will set PIR for uboot, let uboot know EOI has send. > + */ > + out_be32(&spin_table->pir, 0); Why is the kernel sending EOI for an interrupt U-Boot receives? Why are you abusing PIR for anything other than setting the desired PIR value? What happens when you enter the kernel with secondaries having PIR = 0? > + /* Let cpu exit low power state, and from u-boot jump to kernel */ > + arch_send_call_function_single_ipi(kick_cpus); > + > + /* > + * Let we ACK interrput and Send EOI signal to finish INT server > + * U-boot has read EPR to ACK interrput when MPIC work in external > + * proxy mode. Without the external proxy facility, we need to read > + * MPIC ACK register. > + * > + * There just ACK interrput, we don't need to get the interrupt vector > + * and to handle it. Because there just IPI or DOORBELL interrupt to > + * make u-boot exit low power state and jump to kernel. > + */ > + mpic_cpu_ack(nr); How do you know the interrupt is ready to be acked at this point? -Scott
diff --git a/arch/powerpc/include/asm/mpic.h b/arch/powerpc/include/asm/mpic.h index 754f93d..8af6a25 100644 --- a/arch/powerpc/include/asm/mpic.h +++ b/arch/powerpc/include/asm/mpic.h @@ -474,6 +474,15 @@ extern int mpic_cpu_get_priority(void); /* Set the current cpu priority for this cpu */ extern void mpic_cpu_set_priority(int prio); +/* Set cpu priority */ +void mpic_set_cpu_priority(int nr, int prio); + +/* Set cpu EOI */ +void mpic_cpu_eoi_write(int cpu); + +/* CPU ACK interrupt */ +void mpic_cpu_ack(int cpu); + /* Request IPIs on primary mpic */ extern void mpic_request_ipis(void); diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c index d7c1e69..6c54632 100644 --- a/arch/powerpc/platforms/85xx/smp.c +++ b/arch/powerpc/platforms/85xx/smp.c @@ -193,6 +193,30 @@ static int smp_85xx_kick_cpu(int nr) const u64 *cpu_rel_addr; __iomem struct epapr_spin_table *spin_table; struct device_node *np; + + /* + * DOORBELL: + * When kernel kick one of cpus, all cpus will be wakenup. To make + * sure that only the target cpu is effected, other cpus (by checking + * spin_table->addr_l) should go back to low power state. + * + * U-boot has renumber the cpu PIR Why we need to set all of PIR to + * the same value? + * A: Before kernel kicking cpu, the doorbell message was not configured + * for target cpu(cpu_messages->data). If we try to send a + * non-configured message to target cpu, it cannot correctly receive + * doorbell interrput. So SET ALL OF CPU'S PIR to the same value to + * let all cpus catch the interrupt. + * + * Why set PIR to zero? + * A: U-boot cannot know how many cpus will be kicked up(Kernel allow us + * to configure NR_CPUS) and IPI is a per_cpu variable, u-boot cannot + * set a appropriate PIR for every cpu, but the boot cpu(CPU0) always be + * there. U-boot set PIR to zero as a default PIR ID for each CPU, so + * initialize the kick_cpus to 0. + */ + u32 kick_cpus = 0; + int hw_cpu = get_hard_smp_processor_id(nr); int ioremappable; int ret = 0; @@ -251,8 +275,7 @@ static int smp_85xx_kick_cpu(int nr) spin_table = phys_to_virt(*cpu_rel_addr); local_irq_save(flags); -#ifdef CONFIG_PPC32 -#ifdef CONFIG_HOTPLUG_CPU +#if defined(CONFIG_PPC32) && defined(CONFIG_HOTPLUG_CPU) /* Corresponding to generic_set_cpu_dead() */ generic_set_cpu_up(nr); @@ -292,11 +315,58 @@ static int smp_85xx_kick_cpu(int nr) __secondary_hold_acknowledge = -1; } #endif + flush_spin_table(spin_table); - out_be32(&spin_table->pir, hw_cpu); + /* + * U-boot will wait kernel send eoi to MPIC, after EOI has send + * kernel will set PIR for uboot, let uboot know EOI has send. + */ + out_be32(&spin_table->pir, 0); + +#ifdef CONFIG_PPC32 out_be32(&spin_table->addr_l, __pa(__early_start)); +#else + out_be64((u64 *)(&spin_table->addr_h), + __pa(ppc_function_entry(generic_secondary_smp_init))); +#endif flush_spin_table(spin_table); + /* + * e500, e500v2 need to use MPIC to send IPI signal, so we need to + * open IPI firstly. + */ + if (!cpu_has_feature(CPU_FTR_DBELL)) { + mpic_set_cpu_priority(nr, 0); + kick_cpus = nr; + } + + /* Let cpu exit low power state, and from u-boot jump to kernel */ + arch_send_call_function_single_ipi(kick_cpus); + + /* + * Let we ACK interrput and Send EOI signal to finish INT server + * U-boot has read EPR to ACK interrput when MPIC work in external + * proxy mode. Without the external proxy facility, we need to read + * MPIC ACK register. + * + * There just ACK interrput, we don't need to get the interrupt vector + * and to handle it. Because there just IPI or DOORBELL interrupt to + * make u-boot exit low power state and jump to kernel. + */ + mpic_cpu_ack(nr); + /* Send EOI to clear ISR bit to remove interrupt from service */ + mpic_cpu_eoi_write(nr); + + /* After wakeup CPU disable IPI, IPI will be opened in setup_cpu */ + if (!cpu_has_feature(CPU_FTR_DBELL)) + mpic_set_cpu_priority(nr, 0xf); + + /* After EOI finish, let we release cpu */ + flush_spin_table(spin_table); + out_be32(&spin_table->pir, hw_cpu); + flush_spin_table(spin_table); + +#ifdef CONFIG_PPC32 /* Wait a bit for the CPU to ack. */ if (!spin_event_timeout(__secondary_hold_acknowledge == hw_cpu, 10000, 100)) { @@ -308,12 +378,6 @@ static int smp_85xx_kick_cpu(int nr) out: #else smp_generic_kick_cpu(nr); - - flush_spin_table(spin_table); - out_be32(&spin_table->pir, hw_cpu); - out_be64((u64 *)(&spin_table->addr_h), - __pa(ppc_function_entry(generic_secondary_smp_init))); - flush_spin_table(spin_table); #endif local_irq_restore(flags); diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c index c4648ad..b2ba47e 100644 --- a/arch/powerpc/sysdev/mpic.c +++ b/arch/powerpc/sysdev/mpic.c @@ -658,6 +658,21 @@ static inline void mpic_eoi(struct mpic *mpic) (void)mpic_cpu_read(MPIC_INFO(CPU_WHOAMI)); } +void mpic_cpu_eoi_write(int cpu) +{ + struct mpic *mpic = mpic_primary; + + _mpic_write(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_EOI), 0); + _mpic_read(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_WHOAMI)); +} + +void mpic_cpu_ack(int cpu) +{ + struct mpic *mpic = mpic_primary; + + _mpic_read(mpic->reg_type, &mpic->cpuregs[cpu], MPIC_INFO(CPU_INTACK)); +} + /* * Linux descriptor level callbacks */ @@ -1778,6 +1793,16 @@ void mpic_cpu_set_priority(int prio) mpic_cpu_write(MPIC_INFO(CPU_CURRENT_TASK_PRI), prio); } +void mpic_set_cpu_priority(int nr, int prio) +{ + struct mpic *mpic = mpic_primary; + int hw_cpu = get_hard_smp_processor_id(nr); + + prio &= MPIC_CPU_TASKPRI_MASK; + _mpic_write(mpic->reg_type, &mpic->cpuregs[hw_cpu], + MPIC_INFO(CPU_CURRENT_TASK_PRI), prio); +} + void mpic_teardown_this_cpu(int secondary) { struct mpic *mpic = mpic_primary;