| Message ID | 20210416234225.3715819-1-olteanv@gmail.com |
|---|---|
| Headers | show |
| Series | Flow control for NXP ENETC | expand |
>-----Original Message----- >From: Vladimir Oltean <olteanv@gmail.com> >Sent: Saturday, April 17, 2021 2:42 AM >To: Jakub Kicinski <kuba@kernel.org>; David S. Miller ><davem@davemloft.net>; netdev@vger.kernel.org; Po Liu ><po.liu@nxp.com> >Cc: Claudiu Manoil <claudiu.manoil@nxp.com>; Alexandru Marginean ><alexandru.marginean@nxp.com>; Rob Herring <robh+dt@kernel.org>; >Shawn Guo <shawnguo@kernel.org>; linux-arm-kernel@lists.infradead.org; >devicetree@vger.kernel.org; Russell King - ARM Linux admin ><linux@armlinux.org.uk>; Andrew Lunn <andrew@lunn.ch>; Michael Walle ><michael@walle.cc>; Vladimir Oltean <vladimir.oltean@nxp.com> >Subject: [PATCH net-next 5/5] net: enetc: add support for flow control > >From: Vladimir Oltean <vladimir.oltean@nxp.com> > >In the ENETC receive path, a frame received by the MAC is first stored >in a 256KB 'FIFO' memory, then transferred to DRAM when enqueuing it to >the RX ring. The FIFO is a shared resource for all ENETC ports, but >every port keeps track of its own memory utilization, on RX and on TX. > >There is a setting for RX rings through which they can either operate in >'lossy' mode (where the lack of a free buffer causes an immediate >discard of the frame) or in 'lossless' mode (where the lack of a free >buffer in the ring makes the frame stay longer in the FIFO). > >In turn, when the memory utilization of the FIFO exceeds a certain >margin, the MAC can be configured to emit PAUSE frames. > >There is enough FIFO memory to buffer up to 3 MTU-sized frames per RX >port while not jeopardizing the other use cases (jumbo frames), and >also not consume bytes from the port TX allocations. Also, 3 MTU-sized >frames worth of memory is enough to ensure zero loss for 64 byte packets >at 1G line rate. > >Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com> Reviewed-by: Claudiu Manoil <claudiu.manoil@nxp.com>
On Sat, 17 Apr 2021 02:42:20 +0300 Vladimir Oltean wrote: > From: Vladimir Oltean <vladimir.oltean@nxp.com> > > This patch series contains logic for enabling the lossless mode on the > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO > memory. > > During testing it was found that, with the default FIFO configuration, > a sender which isn't persuaded by our PAUSE frames and keeps sending > will cause some MAC RX frame errors. To mitigate this, we need to ensure > that the FIFO never runs completely full, so we need to fix up a setting > that was supposed to be configured well out of reset. Unfortunately this > requires the addition of a new mini-driver. FWIW back in the day when I was working on more advanced devices than I deal with these days I was expecting to eventually run into this as well and create some form of devlink umbrella. IMHO such "mini driver" is a natural place for a devlink instance, and not the PFs/ports. Is this your thinking as well? AFAICT enetc doesn't implement devlink today so you start from whatever model works best without worrying about backward compat.
Hi Jakub, On Mon, Apr 19, 2021 at 02:04:42PM -0700, Jakub Kicinski wrote: > On Sat, 17 Apr 2021 02:42:20 +0300 Vladimir Oltean wrote: > > From: Vladimir Oltean <vladimir.oltean@nxp.com> > > > > This patch series contains logic for enabling the lossless mode on the > > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO > > memory. > > > > During testing it was found that, with the default FIFO configuration, > > a sender which isn't persuaded by our PAUSE frames and keeps sending > > will cause some MAC RX frame errors. To mitigate this, we need to ensure > > that the FIFO never runs completely full, so we need to fix up a setting > > that was supposed to be configured well out of reset. Unfortunately this > > requires the addition of a new mini-driver. > > FWIW back in the day when I was working on more advanced devices than > I deal with these days I was expecting to eventually run into this as > well and create some form of devlink umbrella. IMHO such "mini driver" > is a natural place for a devlink instance, and not the PFs/ports. > Is this your thinking as well? AFAICT enetc doesn't implement devlink > today so you start from whatever model works best without worrying > about backward compat. Sorry, but I am not sure if I understood the central idea of what you were trying to transmit. What is 'a devlink instance and not the PFs'? I am not aware of how a single devlink instance can be exposed for a piece of hardware presenting itself as multiple PFs with multiple driver instances running asynchronously and potentially being assigned to AMP software execution environments (other cores running non-Linux, and most probably Linux is not even the privileged execution environment which has write access to the FIFO parameters). Are you suggesting that the FIFO size and partitioning characteristics be exposed through the devlink subsystem? Isn't that what devlink-sb is for? Also, that would not help with what the IERB driver is trying to achieve. There isn't anything we want the user to view or fiddle with, the reality is simply that the FIFO parameters were supposed to be one-size-fits-all-and-nobody-cares-about-them (the memory usage scheme of this NIC is smart enough to allow for that, or so I think) but nonetheless, the hardware defaults need to be touched up. If LS1028A was a new SoC today we would have probably done this from U-Boot, from the same logic that already passes the MAC addresses to the PFs through the IERB, but the ship has kind of sailed for that, bootloaders are stable, and 'Linux needs this feature' is not a good reason to update them. So this is all that I would like the IERB driver to do, notice how it's all writes of predefined values but no reads. For next generation SoCs with ENETC we'll try our best to not need an IERB driver in Linux at all. Another option would have been to do these fixups in the arch init code as a sort of erratum workaround, but I didn't find a place similar to arch/arm/mach-* for arm64, so I assumed that the arm64 port just doesn't want to go that route. So here I am with a driver for some memory writes.
Hello: This series was applied to netdev/net-next.git (refs/heads/master): On Sat, 17 Apr 2021 02:42:20 +0300 you wrote: > From: Vladimir Oltean <vladimir.oltean@nxp.com> > > This patch series contains logic for enabling the lossless mode on the > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO > memory. > > During testing it was found that, with the default FIFO configuration, > a sender which isn't persuaded by our PAUSE frames and keeps sending > will cause some MAC RX frame errors. To mitigate this, we need to ensure > that the FIFO never runs completely full, so we need to fix up a setting > that was supposed to be configured well out of reset. Unfortunately this > requires the addition of a new mini-driver. > > [...] Here is the summary with links: - [net-next,1/5] net: enetc: create a common enetc_pf_to_port helper https://git.kernel.org/netdev/net-next/c/87614b931c24 - [net-next,2/5] dt-bindings: net: fsl: enetc: add the IERB documentation https://git.kernel.org/netdev/net-next/c/4ac7acc67f29 - [net-next,3/5] net: enetc: add a mini driver for the Integrated Endpoint Register Block https://git.kernel.org/netdev/net-next/c/e7d48e5fbf30 - [net-next,4/5] arm64: dts: ls1028a: declare the Integrated Endpoint Register Block node https://git.kernel.org/netdev/net-next/c/b764dc6cc1ba - [net-next,5/5] net: enetc: add support for flow control https://git.kernel.org/netdev/net-next/c/a8648887880f You are awesome, thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Hi Michael, On Tue, Apr 20, 2021 at 03:27:24PM +0200, Michael Walle wrote: > Hi Vladimir, > > Am 2021-04-17 01:42, schrieb Vladimir Oltean: > > From: Vladimir Oltean <vladimir.oltean@nxp.com> > > > > This patch series contains logic for enabling the lossless mode on the > > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO > > memory. > > > > During testing it was found that, with the default FIFO configuration, > > a sender which isn't persuaded by our PAUSE frames and keeps sending > > will cause some MAC RX frame errors. To mitigate this, we need to ensure > > that the FIFO never runs completely full, so we need to fix up a setting > > that was supposed to be configured well out of reset. Unfortunately this > > requires the addition of a new mini-driver. > > What happens if the mini driver is not enabled? Then the fixes aren't > applied and bad things happen (now with the addition of flow control), > right? > > I'm asking because, if you have the arm64 defconfig its not enabled. > > shouldn't it be something like: > > diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig > b/drivers/net/ethernet/freescale/enetc/Kconfig > index d88f60c2bb82..cdc0ff89388a 100644 > --- a/drivers/net/ethernet/freescale/enetc/Kconfig > +++ b/drivers/net/ethernet/freescale/enetc/Kconfig > @@ -2,7 +2,7 @@ > config FSL_ENETC > tristate "ENETC PF driver" > depends on PCI && PCI_MSI > - depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n > + select FSL_ENETC_IERB > select FSL_ENETC_MDIO > select PHYLINK > select PCS_LYNX Yes, ideally the IERB driver and the ENETC PF driver should be built in the same way, or the IERB driver can be built-in and the PF driver can be module. I don't know how to express this using Kconfig, sorry.
Hi Vladimir, Am 2021-04-20 16:04, schrieb Vladimir Oltean: > On Tue, Apr 20, 2021 at 03:27:24PM +0200, Michael Walle wrote: >> Hi Vladimir, >> >> Am 2021-04-17 01:42, schrieb Vladimir Oltean: >> > From: Vladimir Oltean <vladimir.oltean@nxp.com> >> > >> > This patch series contains logic for enabling the lossless mode on the >> > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO >> > memory. >> > >> > During testing it was found that, with the default FIFO configuration, >> > a sender which isn't persuaded by our PAUSE frames and keeps sending >> > will cause some MAC RX frame errors. To mitigate this, we need to ensure >> > that the FIFO never runs completely full, so we need to fix up a setting >> > that was supposed to be configured well out of reset. Unfortunately this >> > requires the addition of a new mini-driver. >> >> What happens if the mini driver is not enabled? Then the fixes aren't >> applied and bad things happen (now with the addition of flow control), >> right? >> >> I'm asking because, if you have the arm64 defconfig its not enabled. >> >> shouldn't it be something like: >> >> diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig >> b/drivers/net/ethernet/freescale/enetc/Kconfig >> index d88f60c2bb82..cdc0ff89388a 100644 >> --- a/drivers/net/ethernet/freescale/enetc/Kconfig >> +++ b/drivers/net/ethernet/freescale/enetc/Kconfig >> @@ -2,7 +2,7 @@ >> config FSL_ENETC >> tristate "ENETC PF driver" >> depends on PCI && PCI_MSI >> - depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n >> + select FSL_ENETC_IERB >> select FSL_ENETC_MDIO >> select PHYLINK >> select PCS_LYNX > > Yes, ideally the IERB driver and the ENETC PF driver should be built in > the same way, or the IERB driver can be built-in and the PF driver can > be module. I don't know how to express this using Kconfig, sorry. With the small patch above it is: FSL_ENETC=m -> FSL_ENETC_IERB = m or y FSL_ENETC=y -> FSL_ENETC_IERB = y FSL_ENETC=n -> FSL_ENETC_IERB = m,y or n Will you fix it? Should I prepare a patch? -michael
On Tue, Apr 20, 2021 at 04:10:34PM +0200, Michael Walle wrote: > Hi Vladimir, > > Am 2021-04-20 16:04, schrieb Vladimir Oltean: > > On Tue, Apr 20, 2021 at 03:27:24PM +0200, Michael Walle wrote: > > > Hi Vladimir, > > > > > > Am 2021-04-17 01:42, schrieb Vladimir Oltean: > > > > From: Vladimir Oltean <vladimir.oltean@nxp.com> > > > > > > > > This patch series contains logic for enabling the lossless mode on the > > > > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO > > > > memory. > > > > > > > > During testing it was found that, with the default FIFO configuration, > > > > a sender which isn't persuaded by our PAUSE frames and keeps sending > > > > will cause some MAC RX frame errors. To mitigate this, we need to ensure > > > > that the FIFO never runs completely full, so we need to fix up a setting > > > > that was supposed to be configured well out of reset. Unfortunately this > > > > requires the addition of a new mini-driver. > > > > > > What happens if the mini driver is not enabled? Then the fixes aren't > > > applied and bad things happen (now with the addition of flow control), > > > right? > > > > > > I'm asking because, if you have the arm64 defconfig its not enabled. > > > > > > shouldn't it be something like: > > > > > > diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig > > > b/drivers/net/ethernet/freescale/enetc/Kconfig > > > index d88f60c2bb82..cdc0ff89388a 100644 > > > --- a/drivers/net/ethernet/freescale/enetc/Kconfig > > > +++ b/drivers/net/ethernet/freescale/enetc/Kconfig > > > @@ -2,7 +2,7 @@ > > > config FSL_ENETC > > > tristate "ENETC PF driver" > > > depends on PCI && PCI_MSI > > > - depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n > > > + select FSL_ENETC_IERB > > > select FSL_ENETC_MDIO > > > select PHYLINK > > > select PCS_LYNX > > > > Yes, ideally the IERB driver and the ENETC PF driver should be built in > > the same way, or the IERB driver can be built-in and the PF driver can > > be module. I don't know how to express this using Kconfig, sorry. > > With the small patch above it is: > FSL_ENETC=m -> FSL_ENETC_IERB = m or y > FSL_ENETC=y -> FSL_ENETC_IERB = y > FSL_ENETC=n -> FSL_ENETC_IERB = m,y or n > > Will you fix it? Should I prepare a patch? Could you please send the patch? Thanks.
From: Vladimir Oltean <vladimir.oltean@nxp.com> This patch series contains logic for enabling the lossless mode on the RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO memory. During testing it was found that, with the default FIFO configuration, a sender which isn't persuaded by our PAUSE frames and keeps sending will cause some MAC RX frame errors. To mitigate this, we need to ensure that the FIFO never runs completely full, so we need to fix up a setting that was supposed to be configured well out of reset. Unfortunately this requires the addition of a new mini-driver. Vladimir Oltean (5): net: enetc: create a common enetc_pf_to_port helper dt-bindings: net: fsl: enetc: add the IERB documentation net: enetc: add a mini driver for the Integrated Endpoint Register Block arm64: dts: ls1028a: declare the Integrated Endpoint Register Block node net: enetc: add support for flow control .../devicetree/bindings/net/fsl-enetc.txt | 15 ++ .../arm64/boot/dts/freescale/fsl-ls1028a.dtsi | 6 + drivers/net/ethernet/freescale/enetc/Kconfig | 9 + drivers/net/ethernet/freescale/enetc/Makefile | 3 + drivers/net/ethernet/freescale/enetc/enetc.h | 16 ++ .../ethernet/freescale/enetc/enetc_ethtool.c | 18 ++ .../net/ethernet/freescale/enetc/enetc_hw.h | 9 + .../net/ethernet/freescale/enetc/enetc_ierb.c | 155 ++++++++++++++++++ .../net/ethernet/freescale/enetc/enetc_ierb.h | 20 +++ .../net/ethernet/freescale/enetc/enetc_pf.c | 95 ++++++++++- .../net/ethernet/freescale/enetc/enetc_qos.c | 16 +- 11 files changed, 349 insertions(+), 13 deletions(-) create mode 100644 drivers/net/ethernet/freescale/enetc/enetc_ierb.c create mode 100644 drivers/net/ethernet/freescale/enetc/enetc_ierb.h