[v5,0/3] Introduce support for T-head TH1520 Mailbox

The T-head TH1520 SoC supports a hardware mailbox that enables two cores
within the SoC to communicate and coordinate [1]. One example of such
coordination would be cooperation with the T-Head E902 core, which is
responsible for power, clock, and resource management. For example, in
the specific case of the BXM-4-64 GPU, it needs to be powered on by the
E902 core, and the kernel running on the E910 needs to 'ask' the
firmware running on the E902 core to enable power to the GPU island.
Given recent advancements in work on the upstream GPU driver [2], there
is an emerging need to get this code in the mainline kernel.

Link: https://openbeagle.org/beaglev-ahead/beaglev-ahead/-/blob/main/docs/TH1520%20System%20User%20Manual.pdf [1]
Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]

Thanks everyone for taking the time to review this series !

To test this, I've prepared a diff that includes the missing drivers
utilizing the mailbox and enabled the GPU node in the device tree to use
the drm/imagination driver.

I’ve created two branches. Both contain the same code, including this
patch series and some extra commits with mailbox consumers. The only
difference is that one branch has an additional commit that switches ON
calls to OFF calls (and vice versa) to observe the behavior when the GPU
is turned off via the mailbox.

To reproduce, simply clone the repository, add the following extra options
in the config:

CONFIG_THEAD_TH1520_MBOX=y #(this series)
CONFIG_DRM_POWERVR=y
CONFIG_LIGHT_AON=y
CONFIG_LIGHT_AON_PD=y

Then build and deploy on the target.

In case [1], the output should be:
[    2.478394] light_aon_probe: virtual_log_mem=0x000000005faf564a, phy base=0x33600000, size:2097152
[    2.488589] succeed to create power domain debugfs direntry
[    2.494987] powervr ffef400000.gpu: Before reading BVNC
At this point, the system will hang because the driver is trying to read
memory-mapped registers while the GPU isn’t powered on.

In case [2], the GPU powers on correctly, and the BVNC can be read
without issues. If the specific firmware file isn’t available, the
output would look like this:
root@revyos-lpi4a:~# dmesg | grep gpu
[    2.408207] powervr ffef400000.gpu: Before reading BVNC
[    2.413533] powervr ffef400000.gpu: After reading BVNC
[    2.418930] powervr ffef400000.gpu: Direct firmware load for powervr/rogue_36.52.104.182_v1.fw failed with error -2
[    2.429568] powervr ffef400000.gpu: [drm] *ERROR* failed to load firmware powervr/rogue_36.52.104.182_v1.fw (err=-2)
[    2.440403] powervr ffef400000.gpu: probe with driver powervr failed with error -2

Here are the links:
[1] - https://github.com/mwilczy/linux/tree/31_october_demonstrate_mailbox_not_working
[2] - https://github.com/mwilczy/linux/tree/31_october_demonstrate_mailbox_working

Note: U-Boot must load the AON firmware at startup to the address mapped
for AON for this to work properly.

v5:
 - removed 'type' mbox-cell, which isn't really necessary for the use-cases of
   the series, but desribes SW protocol instead of the HW, adjust driver code
   logic to account for this, simplifying it
 - change the register mappings in the device tree node to reflect the manual
 - adjust the driver code to compensate for irregularities in the mappings, do
   this by modifying th1520_map_mmio() with extra argument 'offset'
 - add support for enabling/disabling MBOX clocks in the driver
 - add clocks in the dt-binding/device tree node
 - rebase and retest

v4:
- fixed warning of unused variable
- added Reviewed-by from Krzysztof
- fixed minor cosmetic issues in dt-binding

v3:
- added a comment about mixing devm_ and non-devm resources in the context
  of shared interrupts and explained why it's safe to do so in this
  particular case
- changed the order of resource freeing in the .shutdown callback
- used a wrapper function for register mapping
- since the only conceivable use case for this mailbox driver is
  communication with cores not managed by the kernel, I’ve hard-coded
  this by removing the thead,icu-cpu-id property and adjusted the mailbox
  driver code accordingly.
- added a more detailed description for mbox-cells.
- made some cosmetic changes.
- retested by applying the patch with non-yet-upstreamed patches,
  confirming that the drm/imagination driver can read the registers
  correctly.

v2:
 - fixed thead,th1520-mbox.yaml binding file by dropping redundant
   descriptions, renaming reg-names, removing unnecessary clocks,
   providing constraints and defining ICU's
 - fixed the mailbox driver code to work well with updated binding-file,
   removed clocks support, as it's not necessary for mailbox to work
 - adjusted the device tree node instance of mbox_910t so it will work
   with updated bindings file

Michal Wilczynski (3):
  mailbox: Introduce support for T-head TH1520 Mailbox driver
  dt-bindings: mailbox: Add thead,th1520-mailbox bindings
  riscv: dts: thead: Add mailbox node

 .../bindings/mailbox/thead,th1520-mbox.yaml   |  89 +++
 MAINTAINERS                                   |   2 +
 arch/riscv/boot/dts/thead/th1520.dtsi         |  16 +
 drivers/mailbox/Kconfig                       |  10 +
 drivers/mailbox/Makefile                      |   2 +
 drivers/mailbox/mailbox-th1520.c              | 592 ++++++++++++++++++
 6 files changed, 711 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/mailbox/thead,th1520-mbox.yaml
 create mode 100644 drivers/mailbox/mailbox-th1520.c

Le 31/10/2024 à 21:47, Michal Wilczynski a écrit :
> This driver was tested using the drm/imagination GPU driver. It was able
> to successfully power on the GPU, by passing a command through mailbox
> from E910 core to E902 that's responsible for powering up the GPU. The
> GPU driver was able to read the BVNC version from control registers,
> which confirms it was successfully powered on.
> 
> [   33.957467] powervr ffef400000.gpu: [drm] loaded firmware
> powervr/rogue_36.52.104.182_v1.fw
> [   33.966008] powervr ffef400000.gpu: [drm] FW version v1.0 (build
> 6621747 OS)
> [   38.978542] powervr ffef400000.gpu: [drm] *ERROR* Firmware failed to
> boot
> 
> Though the driver still fails to boot the firmware, the mailbox driver
> works when used with the not-yet-upstreamed firmware AON driver. There
> is ongoing work to get the BXM-4-64 supported with the drm/imagination
> driver [1], though it's not completed yet.
> 
> This work is based on the driver from the vendor kernel [2].
> 
> Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/2 [1]
> Link: https://github.com/revyos/thead-kernel.git [2]
> 
> Signed-off-by: Michal Wilczynski <m.wilczynski@samsung.com>
> ---

Hi,

> +static int th1520_mbox_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct th1520_mbox_priv *priv;
> +	unsigned int remote_idx = 0;
> +	unsigned int i;
> +	int ret;
> +
> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	priv->dev = dev;
> +
> +	priv->clocks[0].id = "clk-local";
> +	priv->clocks[1].id = "clk-remote-icu0";
> +	priv->clocks[2].id = "clk-remote-icu1";
> +	priv->clocks[3].id = "clk-remote-icu2";
> +
> +	ret = devm_clk_bulk_get(dev, ARRAY_SIZE(priv->clocks),
> +				priv->clocks);
> +	if (ret) {
> +		dev_err(dev, "Failed to get clocks\n");
> +		return ret;
> +	}
> +
> +	ret = clk_bulk_prepare_enable(ARRAY_SIZE(priv->clocks), priv->clocks);

This should be undone if an error occurs later in the probe.
Using a devm_action_or_reset() would certainly be nicer and would avoid 
the need for a .remove() function.

> +	if (ret) {
> +		dev_err(dev, "Failed to enable clocks\n");
> +		return ret;
> +	}
> +
> +	/*
> +	 * The address mappings in the device tree align precisely with those
> +	 * outlined in the manual. However, register offsets within these
> +	 * mapped regions are irregular, particularly for remote-icu0.
> +	 * Consequently, th1520_map_mmio() requires an additional parameter to
> +	 * handle this quirk.
> +	 */
> +	priv->local_icu[TH_1520_MBOX_ICU_KERNEL_CPU0] =
> +		th1520_map_mmio(pdev, "local", 0x0);
> +	if (IS_ERR(priv->local_icu[TH_1520_MBOX_ICU_KERNEL_CPU0]))
> +		return PTR_ERR(priv->local_icu[TH_1520_MBOX_ICU_KERNEL_CPU0]);
> +
> +	priv->remote_icu[0] = th1520_map_mmio(pdev, "remote-icu0", 0x4000);
> +	if (IS_ERR(priv->remote_icu[0]))
> +		return PTR_ERR(priv->remote_icu[0]);
> +
> +	priv->remote_icu[1] = th1520_map_mmio(pdev, "remote-icu1", 0x0);
> +	if (IS_ERR(priv->remote_icu[1]))
> +		return PTR_ERR(priv->remote_icu[1]);
> +
> +	priv->remote_icu[2] = th1520_map_mmio(pdev, "remote-icu2", 0x0);
> +	if (IS_ERR(priv->remote_icu[2]))
> +		return PTR_ERR(priv->remote_icu[2]);
> +
> +	priv->local_icu[TH_1520_MBOX_ICU_CPU1] =
> +		priv->local_icu[TH_1520_MBOX_ICU_KERNEL_CPU0] +
> +		TH_1520_MBOX_CHAN_RES_SIZE;
> +	priv->local_icu[TH_1520_MBOX_ICU_CPU2] =
> +		priv->local_icu[TH_1520_MBOX_ICU_CPU1] +
> +		TH_1520_MBOX_CHAN_RES_SIZE;
> +	priv->local_icu[TH_1520_MBOX_ICU_CPU3] =
> +		priv->local_icu[TH_1520_MBOX_ICU_CPU2] +
> +		TH_1520_MBOX_CHAN_RES_SIZE;
> +
> +	priv->cur_cpu_ch_base = priv->local_icu[TH_1520_MBOX_ICU_KERNEL_CPU0];
> +
> +	priv->irq = platform_get_irq(pdev, 0);
> +	if (priv->irq < 0)
> +		return priv->irq;
> +
> +	/* init the chans */
> +	for (i = 0; i < TH_1520_MBOX_CHANS; i++) {
> +		struct th1520_mbox_con_priv *cp = &priv->con_priv[i];
> +
> +		cp->idx = i;
> +		cp->chan = &priv->mbox_chans[i];
> +		priv->mbox_chans[i].con_priv = cp;
> +		snprintf(cp->irq_desc, sizeof(cp->irq_desc),
> +			 "th1520_mbox_chan[%i]", cp->idx);
> +
> +		cp->comm_local_base = priv->local_icu[i];
> +		if (i != TH_1520_MBOX_ICU_KERNEL_CPU0) {
> +			cp->comm_remote_base = priv->remote_icu[remote_idx];
> +			remote_idx++;
> +		}
> +	}
> +
> +	spin_lock_init(&priv->mbox_lock);
> +
> +	priv->mbox.dev = dev;
> +	priv->mbox.ops = &th1520_mbox_ops;
> +	priv->mbox.chans = priv->mbox_chans;
> +	priv->mbox.num_chans = TH_1520_MBOX_CHANS;
> +	priv->mbox.of_xlate = th1520_mbox_xlate;
> +	priv->mbox.txdone_irq = true;
> +
> +	platform_set_drvdata(pdev, priv);
> +
> +	ret = th1520_mbox_init_generic(priv);
> +	if (ret) {
> +		dev_err(dev, "Failed to init mailbox context\n");
> +		return ret;
> +	}
> +
> +	return devm_mbox_controller_register(dev, &priv->mbox);
> +}
> +
> +static void th1520_mbox_remove(struct platform_device *pdev)
> +{
> +	struct th1520_mbox_priv *priv = platform_get_drvdata(pdev);
> +
> +	clk_bulk_disable_unprepare(ARRAY_SIZE(priv->clocks), priv->clocks);
> +}

...

CJ