[V3,1/1] riscv: Increase the upper bound of NR_CPUS to 4095

From: Xiang W <wxjstz@126.com>

The maximum hart number is 32 and is determined by
the type of gd->arch.available_harts. However, various
RISC-V specifications allow the hart number to be greater
than 32. We can eliminate this limitation through bitmaps.

The patch modifies the upper bound of the hart number to 4095,
which is also the maximum hart number RISC-V Advanced Core Local
Interruptor Specification gives.

Some defconfigs, (e.g. qemu-riscv32_smode_defconfig,
qemu-riscv64_smode_defconfig, openpiton_riscv64_defconfig, ...)
do not define CONFIG_NR_CPUS, and may result in compile error,
so define CONFIG_NR_CPUS to be 1 if CONFIG_NR_CPUS is not defined.

Tested on sifive unmatched.

Signed-off-by: Xiang W <wxjstz@126.com>
Signed-off-by: Leo Yu-Chi Liang <ycliang@andestech.com>
---
Changes v2 -> v3:
- Fix the calculation of the GD_AVAILABLE_HARTS start address
- Fix compilation error for defconfigs that do not define CONFIG_NR_CPUS
---
 arch/riscv/Kconfig                   |  4 ++--
 arch/riscv/cpu/start.S               | 20 +++++++++++++++-----
 arch/riscv/include/asm/global_data.h |  8 +++++++-
 arch/riscv/lib/smp.c                 |  2 +-
 4 files changed, 25 insertions(+), 9 deletions(-)

On 2/13/23 09:43, Leo Yu-Chi Liang wrote:
> From: Xiang W <wxjstz@126.com>
> 
> The maximum hart number is 32 and is determined by
> the type of gd->arch.available_harts. However, various
> RISC-V specifications allow the hart number to be greater
> than 32. We can eliminate this limitation through bitmaps.
> 
> The patch modifies the upper bound of the hart number to 4095,
> which is also the maximum hart number RISC-V Advanced Core Local
> Interruptor Specification gives.
> 
> Some defconfigs, (e.g. qemu-riscv32_smode_defconfig,
> qemu-riscv64_smode_defconfig, openpiton_riscv64_defconfig, ...)
> do not define CONFIG_NR_CPUS, and may result in compile error,
> so define CONFIG_NR_CPUS to be 1 if CONFIG_NR_CPUS is not defined.
> 
> Tested on sifive unmatched.
> 
> Signed-off-by: Xiang W <wxjstz@126.com>
> Signed-off-by: Leo Yu-Chi Liang <ycliang@andestech.com>
> ---
> Changes v2 -> v3:
> - Fix the calculation of the GD_AVAILABLE_HARTS start address
> - Fix compilation error for defconfigs that do not define CONFIG_NR_CPUS
> ---
>   arch/riscv/Kconfig                   |  4 ++--
>   arch/riscv/cpu/start.S               | 20 +++++++++++++++-----
>   arch/riscv/include/asm/global_data.h |  8 +++++++-
>   arch/riscv/lib/smp.c                 |  2 +-
>   4 files changed, 25 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index ebc4bef220..063734cbb9 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -232,8 +232,8 @@ config SPL_SMP
>   	  all, single processor machines.
>   
>   config NR_CPUS
> -	int "Maximum number of CPUs (2-32)"
> -	range 2 32
> +	int "Maximum number of CPUs (2-4095)"
> +	range 2 4095

Looking at patch
776e8aca0bad ("riscv: alloc space exhausted")
SYS_MALLOC_F_LEN needs to be increased if the number of CPUs is increased.

You could add an architecture specific rule to config.mk to adjust 
SYS_MALLOC_F_LEN.

>   	depends on SMP || SPL_SMP
>   	default 8
>   	help
> diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
> index 4687bca3c9..ae19c2b43d 100644
> --- a/arch/riscv/cpu/start.S
> +++ b/arch/riscv/cpu/start.S
> @@ -172,11 +172,21 @@ wait_for_gd_init:
>   	bnez	t1, 1b
>   
>   	/* register available harts in the available_harts mask */
> -	li	t1, 1
> -	sll	t1, t1, tp
> -	LREG	t2, GD_AVAILABLE_HARTS(gp)
> -	or	t2, t2, t1
> -	SREG	t2, GD_AVAILABLE_HARTS(gp)
> +	li	t1, GD_AVAILABLE_HARTS
> +	add	t1, t1, gp
> +#if defined(CONFIG_ARCH_RV64I)
> +	srli	t2, tp, 6
> +	slli	t2, t2, 3
> +#elif defined(CONFIG_ARCH_RV32I)
> +	srli	t2, tp, 5
> +	slli	t2, t2, 2
> +#endif

If QEMU provides > XLEN harts, this will result in a buffer overrun. You 
must check against the size of available_harts.

> +	add	t1, t1, t2
> +	LREG	t2, 0(t1)
> +	li	t3, 1
> +	sll	t3, t3, tp
> +	or	t2, t2, t3
> +	SREG	t2, 0(t1)
>   
>   	amoswap.w.rl zero, zero, 0(t0)
>   #endif
> diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h
> index 6fdc86dd8b..7e37e90db0 100644
> --- a/arch/riscv/include/asm/global_data.h
> +++ b/arch/riscv/include/asm/global_data.h
> @@ -10,9 +10,15 @@
>   #ifndef	__ASM_GBL_DATA_H
>   #define __ASM_GBL_DATA_H
>   
> +#include <config.h>
>   #include <asm/smp.h>
>   #include <asm/u-boot.h>
>   #include <compiler.h>
> +#include <linux/bitops.h>
> +
> +#ifndef CONFIG_NR_CPUS
> +#define CONFIG_NR_CPUS 1

CONFIG_ constants should not be defined in C code. Please, move this to 
Kconfig.

The chosen value will create problems for QEMU. The fallback value 
should be XLEN (BITS_PER_LONG) at least.

Best regards

Heinrich

> +#endif
>   
>   /* Architecture-specific global data */
>   struct arch_global_data {
> @@ -29,7 +35,7 @@ struct arch_global_data {
>   #endif
>   #if !CONFIG_IS_ENABLED(XIP)
>   #ifdef CONFIG_AVAILABLE_HARTS
> -	ulong available_harts;
> +	ulong available_harts[BITS_TO_LONGS(CONFIG_NR_CPUS)];
>   #endif
>   #endif
>   };
> diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c
> index 4f073a016f..511e3ed98d 100644
> --- a/arch/riscv/lib/smp.c
> +++ b/arch/riscv/lib/smp.c
> @@ -48,7 +48,7 @@ static int send_ipi_many(struct ipi_data *ipi, int wait)
>   #if !CONFIG_IS_ENABLED(XIP)
>   #ifdef CONFIG_AVAILABLE_HARTS
>   		/* skip if hart is not available */
> -		if (!(gd->arch.available_harts & (1 << reg)))
> +		if (!test_bit(reg, gd->arch.available_harts))
>   			continue;
>   #endif
>   #endif