mbox series

[v8,0/7] Add support for Translation Buffer Units

Message ID 20240417133731.2055383-1-quic_c_gdjako@quicinc.com
Headers show
Series Add support for Translation Buffer Units | expand

Message

Georgi Djakov April 17, 2024, 1:37 p.m. UTC
The TCUs (Translation Control Units) and TBUs (Translation Buffer
Units) are key components of the MMU-500. Multiple TBUs are connected
to a single TCU over an interconnect. Each TBU contains a TLB that
caches page tables. The MMU-500 implements a TBU for each connected
master, and the TBU is designed, so that it is local to the master.
A common TBU DT schema is added to describe the TBUs.

The Qualcomm SDM845 and SC7280 platforms have an implementation of the
SMMU-500, that has multiple TBUs. A vendor-specific DT schema is added
to describe the resources for each TBU (register space, power-domains,
interconnects and clocks).

The TBU driver will manage the resources and allow the system to
operate the TBUs during a context fault to obtain details by doing
s1 inv, software + hardware page table walks etc. This is implemented
with ATOS/eCATs as the ATS feature is not supported. Being able to
query the TBUs is useful for debugging various hardware/software
issues on these platforms.

v8:
- Use ARM_SMMU_QCOM_DEBUG instead of a separate config option (Will)
- Replace mutex/spinlock handling with cleanup.h guard() (Konrad)
- Add missing power-domains in sdm845.dtsi (Konrad)
- Fix swapped labels in sc7280.dtsi (Konrad)

v7: https://lore.kernel.org/r/20240329210638.3647523-1-quic_c_gdjako@quicinc.com
- Pick Reviewed-by for the DT binding (Rob)
- Don't spam the log if the dts changes are not there (Konrad)

v6: https://lore.kernel.org/r/20240307190525.395291-1-quic_c_gdjako@quicinc.com/
- Use SoC-specific compatibles (Krzysztof)
- Use additionalProperties: false (Krzysztof)
- Wrap description text to 80 cols (Krzysztof)

v5: https://lore.kernel.org/r/20240226172218.69486-1-quic_c_gdjako@quicinc.com
- Drop the common TBU bindings and child nodes. These TBU functionalities
  are only Qualcomm specific and not generic. In the unmodified ARM MMU-500
  implementation there are no TBU-specific resources, so just make them
  standalone DT nodes. (Robin)
- The "qcom,stream-id-range" DT property now takes a phandle to the smmu
  and a stream ID range.

v4: https://lore.kernel.org/r/20240201210529.7728-1-quic_c_gdjako@quicinc.com/
- Create a common TBU schema. Move the vendor-specific properties into
  a separate schema that references the common one. (Rob)
- Drop unused DT labels in example, fix regex. (Rob)
- Properly rebase on latest code.

v3: https://lore.kernel.org/r/20231220060236.18600-1-quic_c_gdjako@quicinc.com
- Having a TBU is not Qualcomm specific, so allow having TBU child
  nodes with no specific constraints on properties. For some of the
  vendor compatibles however, add a schema to describe specific
  properties and allow validation. (Rob)
- Drop the useless reg-names DT property on TBUs. (Rob)
- Make the stream-id-range DT property a common one. (Rob)
- Fix the DT example. (Rob)
- Minor fixes on the TBU driver.
- Add support for SC7280 platforms.

v2: https://lore.kernel.org/r/20231118042730.2799-1-quic_c_gdjako@quicinc.com
- Improve DT binding description, add full example. (Konrad)
- Drop Qcom specific stuff from the generic binding. (Rob)
- Unconditionally try to populate subnodes. (Konrad)
- Improve TBU driver commit text, remove memory barriers. (Bjorn)
- Move TBU stuff into separate file. Make the driver builtin.
- TODO: Evaluate whether to keep TBU support as a separate driver
  or just instantiate things from qcom_smmu_impl_init()

v1: https://lore.kernel.org/r/20231019021923.13939-1-quic_c_gdjako@quicinc.com

Georgi Djakov (7):
  dt-bindings: iommu: Add Qualcomm TBU
  iommu/arm-smmu-qcom-debug: Add support for TBUs
  iommu/arm-smmu: Allow using a threaded handler for context interrupts
  iommu/arm-smmu-qcom: Use a custom context fault handler for sdm845
  arm64: dts: qcom: sdm845: Add DT nodes for the TBUs
  iommu/arm-smmu-qcom: Use the custom fault handler on more platforms
  arm64: dts: qcom: sc7280: Add DT nodes for the TBUs

 .../devicetree/bindings/iommu/qcom,tbu.yaml   |  69 +++
 arch/arm64/boot/dts/qcom/sc7280.dtsi          |  89 ++++
 arch/arm64/boot/dts/qcom/sdm845.dtsi          |  73 +++
 drivers/iommu/Kconfig                         |  12 +-
 .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  | 496 ++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    |   8 +
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |   2 +
 drivers/iommu/arm/arm-smmu/arm-smmu.c         |  12 +-
 drivers/iommu/arm/arm-smmu/arm-smmu.h         |   3 +
 9 files changed, 758 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/iommu/qcom,tbu.yaml

Comments

Will Deacon April 19, 2024, 3:33 p.m. UTC | #1
On Wed, 17 Apr 2024 06:37:24 -0700, Georgi Djakov wrote:
> The TCUs (Translation Control Units) and TBUs (Translation Buffer
> Units) are key components of the MMU-500. Multiple TBUs are connected
> to a single TCU over an interconnect. Each TBU contains a TLB that
> caches page tables. The MMU-500 implements a TBU for each connected
> master, and the TBU is designed, so that it is local to the master.
> A common TBU DT schema is added to describe the TBUs.
> 
> [...]

Applied driver and binding updates to will (for-joerg/arm-smmu/updates),
thanks!

[1/7] dt-bindings: iommu: Add Qualcomm TBU
      https://git.kernel.org/will/c/54a75d8f14c5
[2/7] iommu/arm-smmu-qcom-debug: Add support for TBUs
      https://git.kernel.org/will/c/414ecb030870
[3/7] iommu/arm-smmu: Allow using a threaded handler for context interrupts
      https://git.kernel.org/will/c/960be6e10d4f
[4/7] iommu/arm-smmu-qcom: Use a custom context fault handler for sdm845
      https://git.kernel.org/will/c/d374555ef993

[6/7] iommu/arm-smmu-qcom: Use the custom fault handler on more platforms
      https://git.kernel.org/will/c/b8ca7ce709f8

Cheers,
Georgi Djakov May 2, 2024, 4:53 p.m. UTC | #2
Hi Will,

On 1.05.24 17:34, Will Deacon wrote:
> Hi Georgi,
> 
> On Wed, Apr 17, 2024 at 06:37:26AM -0700, Georgi Djakov wrote:
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
>> index bb89d49adf8d..eff7ca94ec8d 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
> 
> ...
> 
>> +static const struct of_device_id qcom_tbu_of_match[] = {
>> +	{ .compatible = "qcom,sc7280-tbu" },
>> +	{ .compatible = "qcom,sdm845-tbu" },
>> +	{ }
>> +};
>> +
>> +static struct platform_driver qcom_tbu_driver = {
>> +	.driver = {
>> +		.name           = "qcom_tbu",
>> +		.of_match_table = qcom_tbu_of_match,
>> +	},
>> +	.probe = qcom_tbu_probe,
>> +};
>> +builtin_platform_driver(qcom_tbu_driver);
> 
> I just noticed that this breaks a modular build of the arm-smmu driver
> because we now have two init functions for the module:
> 
>    ld.lld: error: duplicate symbol: init_module
>    >>> defined at arm-smmu.c
>    >>>            drivers/iommu/arm/arm-smmu/arm-smmu.o:(init_module)
>    >>> defined at arm-smmu-qcom-debug.c
>    >>>            drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.o:(.init.text+0x4)
> 
> I think you should initialise the TBU debug feature by calling into it
> manually from qcom_smmu_impl_init().
> 
> Please can you send a patch to fix that? For now, I'll bodge it so that
> the qcom debug stuff doesn't build as a module (see below).

Š¢hanks for sorting that out and apologies for missing it. We are at rc6 now and
i am afraid that i won't be able to validate my patch until next week (Easter
holidays here). I'll definitely send the rework when I get back.

BR,
Georgi

> 
> Cheers,
> 
> Will
> 
> --->8
> 
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index 032bfd681307..66325210c8c9 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -376,7 +376,7 @@ config ARM_SMMU_QCOM
> 
>   config ARM_SMMU_QCOM_DEBUG
>          bool "ARM SMMU QCOM implementation defined debug support"
> -       depends on ARM_SMMU_QCOM
> +       depends on ARM_SMMU_QCOM=y
>          help
>            Support for implementation specific debug features in ARM SMMU
>            hardware found in QTI platforms. This include support for
>
Bjorn Andersson May 29, 2024, 2:01 a.m. UTC | #3
On Wed, 17 Apr 2024 06:37:24 -0700, Georgi Djakov wrote:
> The TCUs (Translation Control Units) and TBUs (Translation Buffer
> Units) are key components of the MMU-500. Multiple TBUs are connected
> to a single TCU over an interconnect. Each TBU contains a TLB that
> caches page tables. The MMU-500 implements a TBU for each connected
> master, and the TBU is designed, so that it is local to the master.
> A common TBU DT schema is added to describe the TBUs.
> 
> [...]

Applied, thanks!

[5/7] arm64: dts: qcom: sdm845: Add DT nodes for the TBUs
      commit: 7bb38c20f2b64a65423e64e6765bd70a5eadee81
[7/7] arm64: dts: qcom: sc7280: Add DT nodes for the TBUs
      commit: d1f2b41e96f5d1c2241ef3740a5829d2f9979273

Best regards,
Dmitry Baryshkov June 14, 2024, 6:05 p.m. UTC | #4
On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
>
> Add the device-tree nodes for the TBUs (translation buffer units) that
> are present on the sdm845 platforms. The TBUs can be used debug the
> kernel and provide additional information when a context faults occur.
>
> Describe the all registers, clocks, interconnects and power-domain
> resources that are needed for each of the TBUs.
>
> Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>

This patch now prevents interconnect drivers from hitting the sync
state on SDM845.
The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
enabled, which is not a typical case on a normal system:

[   26.209151] qnoc-sdm845 1500000.interconnect: sync_state() pending
due to 150c5000.tbu
[   26.217228] qnoc-sdm845 1620000.interconnect: sync_state() pending
due to 150c5000.tbu
[   26.229926] qnoc-sdm845 1500000.interconnect: sync_state() pending
due to 150c9000.tbu
[   26.238008] qnoc-sdm845 1620000.interconnect: sync_state() pending
due to 150c9000.tbu
[   26.249068] qnoc-sdm845 1740000.interconnect: sync_state() pending
due to 150cd000.tbu
[   26.257127] qnoc-sdm845 1740000.interconnect: sync_state() pending
due to 150d1000.tbu
[   26.265159] qnoc-sdm845 1740000.interconnect: sync_state() pending
due to 150d5000.tbu
[   26.273189] qnoc-sdm845 1500000.interconnect: sync_state() pending
due to 150d9000.tbu
[   26.281206] qnoc-sdm845 1620000.interconnect: sync_state() pending
due to 150d9000.tbu
[   26.289203] qnoc-sdm845 1500000.interconnect: sync_state() pending
due to 150dd000.tbu
[   26.297196] qnoc-sdm845 1620000.interconnect: sync_state() pending
due to 150dd000.tbu
[   26.305201] qnoc-sdm845 1500000.interconnect: sync_state() pending
due to 150e1000.tbu
[   26.313207] qnoc-sdm845 1620000.interconnect: sync_state() pending
due to 150e1000.tbu


> ---
>  arch/arm64/boot/dts/qcom/sdm845.dtsi | 73 ++++++++++++++++++++++++++++
>  1 file changed, 73 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/qcom/sdm845.dtsi b/arch/arm64/boot/dts/qcom/sdm845.dtsi
> index 2f20be99ee7e..fa9403aad96f 100644
> --- a/arch/arm64/boot/dts/qcom/sdm845.dtsi
> +++ b/arch/arm64/boot/dts/qcom/sdm845.dtsi
> @@ -15,6 +15,7 @@
>  #include <dt-bindings/dma/qcom-gpi.h>
>  #include <dt-bindings/firmware/qcom,scm.h>
>  #include <dt-bindings/gpio/gpio.h>
> +#include <dt-bindings/interconnect/qcom,icc.h>
>  #include <dt-bindings/interconnect/qcom,osm-l3.h>
>  #include <dt-bindings/interconnect/qcom,sdm845.h>
>  #include <dt-bindings/interrupt-controller/arm-gic.h>
> @@ -5085,6 +5086,78 @@ apps_smmu: iommu@15000000 {
>                                      <GIC_SPI 343 IRQ_TYPE_LEVEL_HIGH>;
>                 };
>
> +               anoc_1_tbu: tbu@150c5000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150c5000 0x0 0x1000>;
> +                       interconnects = <&system_noc MASTER_GNOC_SNOC QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &config_noc SLAVE_IMEM_CFG QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_AGGRE_NOC_MMU_TBU1_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x0 0x400>;
> +               };
> +
> +               anoc_2_tbu: tbu@150c9000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150c9000 0x0 0x1000>;
> +                       interconnects = <&system_noc MASTER_GNOC_SNOC QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &config_noc SLAVE_IMEM_CFG QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_AGGRE_NOC_MMU_TBU2_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x400 0x400>;
> +               };
> +
> +               mnoc_hf_0_tbu: tbu@150cd000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150cd000 0x0 0x1000>;
> +                       interconnects = <&mmss_noc MASTER_MDP0 QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &mmss_noc SLAVE_MNOC_HF_MEM_NOC QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_MMNOC_MMU_TBU_HF0_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x800 0x400>;
> +               };
> +
> +               mnoc_hf_1_tbu: tbu@150d1000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150d1000 0x0 0x1000>;
> +                       interconnects = <&mmss_noc MASTER_MDP0 QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &mmss_noc SLAVE_MNOC_HF_MEM_NOC QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_MMNOC_MMU_TBU_HF1_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0xc00 0x400>;
> +               };
> +
> +               mnoc_sf_0_tbu: tbu@150d5000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150d5000 0x0 0x1000>;
> +                       interconnects = <&mmss_noc MASTER_CAMNOC_SF QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &mmss_noc SLAVE_MNOC_SF_MEM_NOC QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_MMNOC_MMU_TBU_SF_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x1000 0x400>;
> +               };
> +
> +               compute_dsp_tbu: tbu@150d9000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150d9000 0x0 0x1000>;
> +                       interconnects = <&system_noc MASTER_GNOC_SNOC QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &config_noc SLAVE_IMEM_CFG QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       qcom,stream-id-range = <&apps_smmu 0x1400 0x400>;
> +               };
> +
> +               adsp_tbu: tbu@150dd000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150dd000 0x0 0x1000>;
> +                       interconnects = <&system_noc MASTER_GNOC_SNOC QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &config_noc SLAVE_IMEM_CFG QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_AGGRE_NOC_MMU_AUDIO_TBU_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x1800 0x400>;
> +               };
> +
> +               anoc_1_pcie_tbu: tbu@150e1000 {
> +                       compatible = "qcom,sdm845-tbu";
> +                       reg = <0x0 0x150e1000 0x0 0x1000>;
> +                       clocks = <&gcc GCC_AGGRE_NOC_PCIE_TBU_CLK>;
> +                       interconnects = <&system_noc MASTER_GNOC_SNOC QCOM_ICC_TAG_ACTIVE_ONLY
> +                                        &config_noc SLAVE_IMEM_CFG QCOM_ICC_TAG_ACTIVE_ONLY>;
> +                       power-domains = <&gcc HLOS1_VOTE_AGGRE_NOC_MMU_PCIE_TBU_GDSC>;
> +                       qcom,stream-id-range = <&apps_smmu 0x1c00 0x400>;
> +               };
> +
>                 lpasscc: clock-controller@17014000 {
>                         compatible = "qcom,sdm845-lpasscc";
>                         reg = <0 0x17014000 0 0x1f004>, <0 0x17300000 0 0x200>;
>
Dmitry Baryshkov June 25, 2024, 7:50 a.m. UTC | #5
On Fri, 14 Jun 2024 at 21:05, Dmitry Baryshkov
<dmitry.baryshkov@linaro.org> wrote:
>
> On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
> >
> > Add the device-tree nodes for the TBUs (translation buffer units) that
> > are present on the sdm845 platforms. The TBUs can be used debug the
> > kernel and provide additional information when a context faults occur.
> >
> > Describe the all registers, clocks, interconnects and power-domain
> > resources that are needed for each of the TBUs.
> >
> > Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
>
> This patch now prevents interconnect drivers from hitting the sync
> state on SDM845.
> The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
> enabled, which is not a typical case on a normal system:

Georgi, before I start acting like a bull in a china shop and sending
reverts, any update from your side?

>
> [   26.209151] qnoc-sdm845 1500000.interconnect: sync_state() pending
> due to 150c5000.tbu
> [   26.217228] qnoc-sdm845 1620000.interconnect: sync_state() pending
> due to 150c5000.tbu
> [   26.229926] qnoc-sdm845 1500000.interconnect: sync_state() pending
> due to 150c9000.tbu
> [   26.238008] qnoc-sdm845 1620000.interconnect: sync_state() pending
> due to 150c9000.tbu
> [   26.249068] qnoc-sdm845 1740000.interconnect: sync_state() pending
> due to 150cd000.tbu
> [   26.257127] qnoc-sdm845 1740000.interconnect: sync_state() pending
> due to 150d1000.tbu
> [   26.265159] qnoc-sdm845 1740000.interconnect: sync_state() pending
> due to 150d5000.tbu
> [   26.273189] qnoc-sdm845 1500000.interconnect: sync_state() pending
> due to 150d9000.tbu
> [   26.281206] qnoc-sdm845 1620000.interconnect: sync_state() pending
> due to 150d9000.tbu
> [   26.289203] qnoc-sdm845 1500000.interconnect: sync_state() pending
> due to 150dd000.tbu
> [   26.297196] qnoc-sdm845 1620000.interconnect: sync_state() pending
> due to 150dd000.tbu
> [   26.305201] qnoc-sdm845 1500000.interconnect: sync_state() pending
> due to 150e1000.tbu
> [   26.313207] qnoc-sdm845 1620000.interconnect: sync_state() pending
> due to 150e1000.tbu
Georgi Djakov June 25, 2024, 12:57 p.m. UTC | #6
On 25.06.24 10:50, Dmitry Baryshkov wrote:
> On Fri, 14 Jun 2024 at 21:05, Dmitry Baryshkov
> <dmitry.baryshkov@linaro.org> wrote:
>>
>> On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
>>>
>>> Add the device-tree nodes for the TBUs (translation buffer units) that
>>> are present on the sdm845 platforms. The TBUs can be used debug the
>>> kernel and provide additional information when a context faults occur.
>>>
>>> Describe the all registers, clocks, interconnects and power-domain
>>> resources that are needed for each of the TBUs.
>>>
>>> Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
>>
>> This patch now prevents interconnect drivers from hitting the sync
>> state on SDM845.
>> The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
>> enabled, which is not a typical case on a normal system:
> 
> Georgi, before I start acting like a bull in a china shop and sending
> reverts, any update from your side?

Hi Dmitry!
Thanks for the report! We can easily add status = "disabled" to the DT
nodes, but please give me some time to take a look what would be the best
way to handle this, as i was out last week and now i am still catching up.

BR,
Georgi

> 
>>
>> [   26.209151] qnoc-sdm845 1500000.interconnect: sync_state() pending
>> due to 150c5000.tbu
>> [   26.217228] qnoc-sdm845 1620000.interconnect: sync_state() pending
>> due to 150c5000.tbu
>> [   26.229926] qnoc-sdm845 1500000.interconnect: sync_state() pending
>> due to 150c9000.tbu
>> [   26.238008] qnoc-sdm845 1620000.interconnect: sync_state() pending
>> due to 150c9000.tbu
>> [   26.249068] qnoc-sdm845 1740000.interconnect: sync_state() pending
>> due to 150cd000.tbu
>> [   26.257127] qnoc-sdm845 1740000.interconnect: sync_state() pending
>> due to 150d1000.tbu
>> [   26.265159] qnoc-sdm845 1740000.interconnect: sync_state() pending
>> due to 150d5000.tbu
>> [   26.273189] qnoc-sdm845 1500000.interconnect: sync_state() pending
>> due to 150d9000.tbu
>> [   26.281206] qnoc-sdm845 1620000.interconnect: sync_state() pending
>> due to 150d9000.tbu
>> [   26.289203] qnoc-sdm845 1500000.interconnect: sync_state() pending
>> due to 150dd000.tbu
>> [   26.297196] qnoc-sdm845 1620000.interconnect: sync_state() pending
>> due to 150dd000.tbu
>> [   26.305201] qnoc-sdm845 1500000.interconnect: sync_state() pending
>> due to 150e1000.tbu
>> [   26.313207] qnoc-sdm845 1620000.interconnect: sync_state() pending
>> due to 150e1000.tbu
>
Dmitry Baryshkov June 25, 2024, 12:59 p.m. UTC | #7
On Tue, 25 Jun 2024 at 15:57, Georgi Djakov <djakov@kernel.org> wrote:
>
> On 25.06.24 10:50, Dmitry Baryshkov wrote:
> > On Fri, 14 Jun 2024 at 21:05, Dmitry Baryshkov
> > <dmitry.baryshkov@linaro.org> wrote:
> >>
> >> On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
> >>>
> >>> Add the device-tree nodes for the TBUs (translation buffer units) that
> >>> are present on the sdm845 platforms. The TBUs can be used debug the
> >>> kernel and provide additional information when a context faults occur.
> >>>
> >>> Describe the all registers, clocks, interconnects and power-domain
> >>> resources that are needed for each of the TBUs.
> >>>
> >>> Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
> >>
> >> This patch now prevents interconnect drivers from hitting the sync
> >> state on SDM845.
> >> The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
> >> enabled, which is not a typical case on a normal system:
> >
> > Georgi, before I start acting like a bull in a china shop and sending
> > reverts, any update from your side?
>
> Hi Dmitry!
> Thanks for the report! We can easily add status = "disabled" to the DT
> nodes, but please give me some time to take a look what would be the best
> way to handle this, as i was out last week and now i am still catching up.

I think the simplest thing would be to move the TBU driver to the
arm-qcom-smmu.c instead of having it in the -debug.c

>
> BR,
> Georgi
>
> >
> >>
> >> [   26.209151] qnoc-sdm845 1500000.interconnect: sync_state() pending
> >> due to 150c5000.tbu
> >> [   26.217228] qnoc-sdm845 1620000.interconnect: sync_state() pending
> >> due to 150c5000.tbu
> >> [   26.229926] qnoc-sdm845 1500000.interconnect: sync_state() pending
> >> due to 150c9000.tbu
> >> [   26.238008] qnoc-sdm845 1620000.interconnect: sync_state() pending
> >> due to 150c9000.tbu
> >> [   26.249068] qnoc-sdm845 1740000.interconnect: sync_state() pending
> >> due to 150cd000.tbu
> >> [   26.257127] qnoc-sdm845 1740000.interconnect: sync_state() pending
> >> due to 150d1000.tbu
> >> [   26.265159] qnoc-sdm845 1740000.interconnect: sync_state() pending
> >> due to 150d5000.tbu
> >> [   26.273189] qnoc-sdm845 1500000.interconnect: sync_state() pending
> >> due to 150d9000.tbu
> >> [   26.281206] qnoc-sdm845 1620000.interconnect: sync_state() pending
> >> due to 150d9000.tbu
> >> [   26.289203] qnoc-sdm845 1500000.interconnect: sync_state() pending
> >> due to 150dd000.tbu
> >> [   26.297196] qnoc-sdm845 1620000.interconnect: sync_state() pending
> >> due to 150dd000.tbu
> >> [   26.305201] qnoc-sdm845 1500000.interconnect: sync_state() pending
> >> due to 150e1000.tbu
> >> [   26.313207] qnoc-sdm845 1620000.interconnect: sync_state() pending
> >> due to 150e1000.tbu
> >
>
Will Deacon July 2, 2024, 4:39 p.m. UTC | #8
On Tue, Jun 25, 2024 at 03:59:27PM +0300, Dmitry Baryshkov wrote:
> On Tue, 25 Jun 2024 at 15:57, Georgi Djakov <djakov@kernel.org> wrote:
> >
> > On 25.06.24 10:50, Dmitry Baryshkov wrote:
> > > On Fri, 14 Jun 2024 at 21:05, Dmitry Baryshkov
> > > <dmitry.baryshkov@linaro.org> wrote:
> > >>
> > >> On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
> > >>>
> > >>> Add the device-tree nodes for the TBUs (translation buffer units) that
> > >>> are present on the sdm845 platforms. The TBUs can be used debug the
> > >>> kernel and provide additional information when a context faults occur.
> > >>>
> > >>> Describe the all registers, clocks, interconnects and power-domain
> > >>> resources that are needed for each of the TBUs.
> > >>>
> > >>> Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
> > >>
> > >> This patch now prevents interconnect drivers from hitting the sync
> > >> state on SDM845.
> > >> The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
> > >> enabled, which is not a typical case on a normal system:
> > >
> > > Georgi, before I start acting like a bull in a china shop and sending
> > > reverts, any update from your side?
> >
> > Hi Dmitry!
> > Thanks for the report! We can easily add status = "disabled" to the DT
> > nodes, but please give me some time to take a look what would be the best
> > way to handle this, as i was out last week and now i am still catching up.
> 
> I think the simplest thing would be to move the TBU driver to the
> arm-qcom-smmu.c instead of having it in the -debug.c

The TBUs aren't used for anything other than debugging, so I'd really
rather they live with the debug code.

Will
Dmitry Baryshkov July 3, 2024, 9:54 a.m. UTC | #9
On Tue, 2 Jul 2024 at 19:39, Will Deacon <will@kernel.org> wrote:
>
> On Tue, Jun 25, 2024 at 03:59:27PM +0300, Dmitry Baryshkov wrote:
> > On Tue, 25 Jun 2024 at 15:57, Georgi Djakov <djakov@kernel.org> wrote:
> > >
> > > On 25.06.24 10:50, Dmitry Baryshkov wrote:
> > > > On Fri, 14 Jun 2024 at 21:05, Dmitry Baryshkov
> > > > <dmitry.baryshkov@linaro.org> wrote:
> > > >>
> > > >> On Wed, 17 Apr 2024 at 16:39, Georgi Djakov <quic_c_gdjako@quicinc.com> wrote:
> > > >>>
> > > >>> Add the device-tree nodes for the TBUs (translation buffer units) that
> > > >>> are present on the sdm845 platforms. The TBUs can be used debug the
> > > >>> kernel and provide additional information when a context faults occur.
> > > >>>
> > > >>> Describe the all registers, clocks, interconnects and power-domain
> > > >>> resources that are needed for each of the TBUs.
> > > >>>
> > > >>> Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
> > > >>
> > > >> This patch now prevents interconnect drivers from hitting the sync
> > > >> state on SDM845.
> > > >> The TBU driver is enabled only when the ARM_SMMU_QCOM_DEBUG is
> > > >> enabled, which is not a typical case on a normal system:
> > > >
> > > > Georgi, before I start acting like a bull in a china shop and sending
> > > > reverts, any update from your side?
> > >
> > > Hi Dmitry!
> > > Thanks for the report! We can easily add status = "disabled" to the DT
> > > nodes, but please give me some time to take a look what would be the best
> > > way to handle this, as i was out last week and now i am still catching up.
> >
> > I think the simplest thing would be to move the TBU driver to the
> > arm-qcom-smmu.c instead of having it in the -debug.c
>
> The TBUs aren't used for anything other than debugging, so I'd really
> rather they live with the debug code.

The problem is that not having any driver bound to the TBU devices
prevents interconnect drivers from hitting the sync state (and thus
lowering the bandwidth to the requested values).
Being that late in the development cycle, I think we should fix the
issue in the fastest way. So I'm close to sending a revert of the DT
changes. Setting status = "disabled" doesn't qualify as a logical
change as having TBU units enabled on a platform-to-platform basis
doesn't make sense.