| Message ID | 20241107084455.3623576-1-hongxing.zhu@nxp.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v1] PCI: dwc: Clean up some unnecessary codes in dw_pcie_suspend_noirq() | expand |
On 11/7/2024 2:14 PM, Richard Zhu wrote: > Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's safe > to send PME_TURN_OFF message regardless of whether the link is up or > down. So, there would be no need to test the LTSSM stat before sending > PME_TURN_OFF message. > > Remove the L2 poll too, after the PME_TURN_OFF message is sent out. > Because the re-initialization would be done in dw_pcie_resume_noirq(). > we should not remove the poll here, it is required for the endpoint to go gracefully in to L2. Some endpoints can have some cleanups needs to be done before entering into L2 or L3. For the PME turnoff message, the endpoints needs to send L23 ack which indicates endpoint is ready to L2 without that it will not be gracefull D3cold sequence. -Krishna Chaitanya. > Signed-off-by: Richard Zhu <hongxing.zhu@nxp.com> > --- > .../pci/controller/dwc/pcie-designware-host.c | 20 ++++--------------- > 1 file changed, 4 insertions(+), 16 deletions(-) > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > index f86347452026..64c49adf81d2 100644 > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > @@ -917,7 +917,6 @@ static int dw_pcie_pme_turn_off(struct dw_pcie *pci) > int dw_pcie_suspend_noirq(struct dw_pcie *pci) > { > u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > - u32 val; > int ret = 0; > > /* > @@ -927,23 +926,12 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) > if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) > return 0; > > - /* Only send out PME_TURN_OFF when PCIE link is up */ > - if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { > - if (pci->pp.ops->pme_turn_off) > - pci->pp.ops->pme_turn_off(&pci->pp); > - else > - ret = dw_pcie_pme_turn_off(pci); > - > + if (pci->pp.ops->pme_turn_off) { > + pci->pp.ops->pme_turn_off(&pci->pp); > + } else { > + ret = dw_pcie_pme_turn_off(pci); > if (ret) > return ret; > - > - ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, > - PCIE_PME_TO_L2_TIMEOUT_US/10, > - PCIE_PME_TO_L2_TIMEOUT_US, false, pci); > - if (ret) { > - dev_err(pci->dev, "Timeout waiting for L2 entry! LTSSM: 0x%x\n", val); > - return ret; > - }> } > > dw_pcie_stop_link(pci);
On Thu, Nov 07, 2024 at 04:44:55PM +0800, Richard Zhu wrote: > Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's safe > to send PME_TURN_OFF message regardless of whether the link is up or > down. So, there would be no need to test the LTSSM stat before sending > PME_TURN_OFF message. > What is the incentive to send PME_Turn_Off when link is not up? > Remove the L2 poll too, after the PME_TURN_OFF message is sent out. > Because the re-initialization would be done in dw_pcie_resume_noirq(). > As Krishna explained, host needs to wait until the endpoint acks the message (just to give it some time to do cleanups). Then only the host can initiate D3Cold. It matters when the device supports L2. - Mani > Signed-off-by: Richard Zhu <hongxing.zhu@nxp.com> > --- > .../pci/controller/dwc/pcie-designware-host.c | 20 ++++--------------- > 1 file changed, 4 insertions(+), 16 deletions(-) > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > index f86347452026..64c49adf81d2 100644 > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > @@ -917,7 +917,6 @@ static int dw_pcie_pme_turn_off(struct dw_pcie *pci) > int dw_pcie_suspend_noirq(struct dw_pcie *pci) > { > u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > - u32 val; > int ret = 0; > > /* > @@ -927,23 +926,12 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) > if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) > return 0; > > - /* Only send out PME_TURN_OFF when PCIE link is up */ > - if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { > - if (pci->pp.ops->pme_turn_off) > - pci->pp.ops->pme_turn_off(&pci->pp); > - else > - ret = dw_pcie_pme_turn_off(pci); > - > + if (pci->pp.ops->pme_turn_off) { > + pci->pp.ops->pme_turn_off(&pci->pp); > + } else { > + ret = dw_pcie_pme_turn_off(pci); > if (ret) > return ret; > - > - ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, > - PCIE_PME_TO_L2_TIMEOUT_US/10, > - PCIE_PME_TO_L2_TIMEOUT_US, false, pci); > - if (ret) { > - dev_err(pci->dev, "Timeout waiting for L2 entry! LTSSM: 0x%x\n", val); > - return ret; > - } > } > > dw_pcie_stop_link(pci); > -- > 2.37.1 >
On Thu, Nov 07, 2024 at 11:13:34AM +0000, Manivannan Sadhasivam wrote: > On Thu, Nov 07, 2024 at 04:44:55PM +0800, Richard Zhu wrote: > > Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's safe > > to send PME_TURN_OFF message regardless of whether the link is up or > > down. So, there would be no need to test the LTSSM stat before sending > > PME_TURN_OFF message. > > > > What is the incentive to send PME_Turn_Off when link is not up? see Bjorn's comments in https://lore.kernel.org/imx/20241106222933.GA1543549@bhelgaas/ "But I don't think you responded to the race question. What happens here? if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { --> link goes down here <-- pci->pp.ops->pme_turn_off(&pci->pp); You decide the LTSSM is active and the link is up. Then the link goes down. Then you send PME_Turn_off. Now what? If it's safe to try to send PME_Turn_off regardless of whether the link is up or down, there would be no need to test the LTSSM state." I think it may happen if EP device HOT remove/reset after if check. Frank > > > Remove the L2 poll too, after the PME_TURN_OFF message is sent out. > > Because the re-initialization would be done in dw_pcie_resume_noirq(). > > > > As Krishna explained, host needs to wait until the endpoint acks the message > (just to give it some time to do cleanups). Then only the host can initiate > D3Cold. It matters when the device supports L2. > > - Mani > > > Signed-off-by: Richard Zhu <hongxing.zhu@nxp.com> > > --- > > .../pci/controller/dwc/pcie-designware-host.c | 20 ++++--------------- > > 1 file changed, 4 insertions(+), 16 deletions(-) > > > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > > index f86347452026..64c49adf81d2 100644 > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > > @@ -917,7 +917,6 @@ static int dw_pcie_pme_turn_off(struct dw_pcie *pci) > > int dw_pcie_suspend_noirq(struct dw_pcie *pci) > > { > > u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > > - u32 val; > > int ret = 0; > > > > /* > > @@ -927,23 +926,12 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) > > if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) > > return 0; > > > > - /* Only send out PME_TURN_OFF when PCIE link is up */ > > - if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { > > - if (pci->pp.ops->pme_turn_off) > > - pci->pp.ops->pme_turn_off(&pci->pp); > > - else > > - ret = dw_pcie_pme_turn_off(pci); > > - > > + if (pci->pp.ops->pme_turn_off) { > > + pci->pp.ops->pme_turn_off(&pci->pp); > > + } else { > > + ret = dw_pcie_pme_turn_off(pci); > > if (ret) > > return ret; > > - > > - ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, > > - PCIE_PME_TO_L2_TIMEOUT_US/10, > > - PCIE_PME_TO_L2_TIMEOUT_US, false, pci); > > - if (ret) { > > - dev_err(pci->dev, "Timeout waiting for L2 entry! LTSSM: 0x%x\n", val); > > - return ret; > > - } > > } > > > > dw_pcie_stop_link(pci); > > -- > > 2.37.1 > > > > -- > மணிவண்ணன் சதாசிவம்
On Thu, Nov 07, 2024 at 11:08:37AM -0500, Frank Li wrote: > On Thu, Nov 07, 2024 at 11:13:34AM +0000, Manivannan Sadhasivam wrote: > > On Thu, Nov 07, 2024 at 04:44:55PM +0800, Richard Zhu wrote: > > > Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's safe > > > to send PME_TURN_OFF message regardless of whether the link is up or > > > down. So, there would be no need to test the LTSSM stat before sending > > > PME_TURN_OFF message. > > > > > > > What is the incentive to send PME_Turn_Off when link is not up? > > see Bjorn's comments in https://lore.kernel.org/imx/20241106222933.GA1543549@bhelgaas/ > Thanks for the pointer. Let me reply there itsef. - Mani > "But I don't think you responded to the race question. What happens > here? > > if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { > --> link goes down here <-- > pci->pp.ops->pme_turn_off(&pci->pp); > > You decide the LTSSM is active and the link is up. Then the link goes > down. Then you send PME_Turn_off. Now what? > > If it's safe to try to send PME_Turn_off regardless of whether the > link is up or down, there would be no need to test the LTSSM state." > > I think it may happen if EP device HOT remove/reset after if check. > > Frank > > > > > Remove the L2 poll too, after the PME_TURN_OFF message is sent out. > > > Because the re-initialization would be done in dw_pcie_resume_noirq(). > > > > > > > As Krishna explained, host needs to wait until the endpoint acks the message > > (just to give it some time to do cleanups). Then only the host can initiate > > D3Cold. It matters when the device supports L2. > > > > - Mani > > > > > Signed-off-by: Richard Zhu <hongxing.zhu@nxp.com> > > > --- > > > .../pci/controller/dwc/pcie-designware-host.c | 20 ++++--------------- > > > 1 file changed, 4 insertions(+), 16 deletions(-) > > > > > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > > > index f86347452026..64c49adf81d2 100644 > > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > > > @@ -917,7 +917,6 @@ static int dw_pcie_pme_turn_off(struct dw_pcie *pci) > > > int dw_pcie_suspend_noirq(struct dw_pcie *pci) > > > { > > > u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); > > > - u32 val; > > > int ret = 0; > > > > > > /* > > > @@ -927,23 +926,12 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) > > > if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) > > > return 0; > > > > > > - /* Only send out PME_TURN_OFF when PCIE link is up */ > > > - if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { > > > - if (pci->pp.ops->pme_turn_off) > > > - pci->pp.ops->pme_turn_off(&pci->pp); > > > - else > > > - ret = dw_pcie_pme_turn_off(pci); > > > - > > > + if (pci->pp.ops->pme_turn_off) { > > > + pci->pp.ops->pme_turn_off(&pci->pp); > > > + } else { > > > + ret = dw_pcie_pme_turn_off(pci); > > > if (ret) > > > return ret; > > > - > > > - ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, > > > - PCIE_PME_TO_L2_TIMEOUT_US/10, > > > - PCIE_PME_TO_L2_TIMEOUT_US, false, pci); > > > - if (ret) { > > > - dev_err(pci->dev, "Timeout waiting for L2 entry! LTSSM: 0x%x\n", val); > > > - return ret; > > > - } > > > } > > > > > > dw_pcie_stop_link(pci); > > > -- > > > 2.37.1 > > > > > > > -- > > மணிவண்ணன் சதாசிவம்
On Thu, Nov 07, 2024 at 11:13:34AM +0000, Manivannan Sadhasivam wrote: > On Thu, Nov 07, 2024 at 04:44:55PM +0800, Richard Zhu wrote: > > Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's > > safe to send PME_TURN_OFF message regardless of whether the link > > is up or down. So, there would be no need to test the LTSSM stat > > before sending PME_TURN_OFF message. > > What is the incentive to send PME_Turn_Off when link is not up? There's no need to send PME_Turn_Off when link is not up. But a link-up check is inherently racy because the link may go down between the check and the PME_Turn_Off. Since it's impossible for software to guarantee the link is up, the Root Port should be able to tolerate attempts to send PME_Turn_Off when the link is down. So IMO there's no need to check whether the link is up, and checking gives the misleading impression that "we know the link is up and therefore sending PME_Turn_Off is safe." > > Remove the L2 poll too, after the PME_TURN_OFF message is sent > > out. Because the re-initialization would be done in > > dw_pcie_resume_noirq(). > > As Krishna explained, host needs to wait until the endpoint acks the > message (just to give it some time to do cleanups). Then only the > host can initiate D3Cold. It matters when the device supports L2. The important thing here is to be clear about the *reason* to poll for L2 and the *event* that must wait for L2. I don't have any DesignWare specs, but when dw_pcie_suspend_noirq() waits for DW_PCIE_LTSSM_L2_IDLE, I think what we're doing is waiting for the link to be in the L2/L3 Ready pseudo-state (PCIe r6.0, sec 5.2, fig 5-1). L2 and L3 are states where main power to the downstream component is off, i.e., the component is in D3cold (r6.0, sec 5.3.2), so there is no link in those states. The PME_Turn_Off handshake is part of the process to put the downstream component in D3cold. I think the reason for this handshake is to allow an orderly shutdown of that component before main power is removed. When the downstream component receives PME_Turn_Off, it will stop scheduling new TLPs, but it may already have TLPs scheduled but not yet sent. If power were removed immediately, they would be lost. My understanding is that the link will not enter L2/L3 Ready until the components on both ends have completed whatever needs to be done with those TLPs. (This is based on the L2/L3 discussion in the Mindshare PCIe book; I haven't found clear spec citations for all of it.) I think waiting for L2/L3 Ready is to keep us from turning off main power when the components are still trying to dispose of those TLPs. So I think every controller that turns off main power needs to wait for L2/L3 Ready. There's also a requirement that software wait at least 100 ns after L2/L3 Ready before turning off refclock and main power (sec 5.3.3.2.1). Bjorn
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index f86347452026..64c49adf81d2 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -917,7 +917,6 @@ static int dw_pcie_pme_turn_off(struct dw_pcie *pci) int dw_pcie_suspend_noirq(struct dw_pcie *pci) { u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); - u32 val; int ret = 0; /* @@ -927,23 +926,12 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) return 0; - /* Only send out PME_TURN_OFF when PCIE link is up */ - if (dw_pcie_get_ltssm(pci) > DW_PCIE_LTSSM_DETECT_ACT) { - if (pci->pp.ops->pme_turn_off) - pci->pp.ops->pme_turn_off(&pci->pp); - else - ret = dw_pcie_pme_turn_off(pci); - + if (pci->pp.ops->pme_turn_off) { + pci->pp.ops->pme_turn_off(&pci->pp); + } else { + ret = dw_pcie_pme_turn_off(pci); if (ret) return ret; - - ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, - PCIE_PME_TO_L2_TIMEOUT_US/10, - PCIE_PME_TO_L2_TIMEOUT_US, false, pci); - if (ret) { - dev_err(pci->dev, "Timeout waiting for L2 entry! LTSSM: 0x%x\n", val); - return ret; - } } dw_pcie_stop_link(pci);
Before sending PME_TURN_OFF, don't test the LTSSM stat. Since it's safe to send PME_TURN_OFF message regardless of whether the link is up or down. So, there would be no need to test the LTSSM stat before sending PME_TURN_OFF message. Remove the L2 poll too, after the PME_TURN_OFF message is sent out. Because the re-initialization would be done in dw_pcie_resume_noirq(). Signed-off-by: Richard Zhu <hongxing.zhu@nxp.com> --- .../pci/controller/dwc/pcie-designware-host.c | 20 ++++--------------- 1 file changed, 4 insertions(+), 16 deletions(-)