Message ID | 20190328132742.12070-1-leon@kernel.org |
---|---|
Headers | show |
Series | Move IB representors to single IB device multiple ports | expand |
On Thu, Mar 28, 2019 at 03:27:30PM +0200, Leon Romanovsky wrote: > From: Leon Romanovsky <leonro@mellanox.com> > > >From Mark, > > Hi, > > This series starts from two refactoring patches performed in mlx5_core > and they are part of a larger series intended to expose the ability > to insert FDB steering rules by the RDMA side. > > They are followed by patches which move IB representors to single IB > device. Currently when moving to switchdev mode an IB device is created > per virtual function (and one for the uplink). While initially this was > right approach, the limitation of this model requires us to change it. > > Such limitations includes: > - Inability to share resources between IB devices. > - Lack of DEVX to allow more control and device specific functionality. > - Interacting with the E-Switch. > - Doesn't work well with namespaces. > > In this series we move to a single IB device with multiple ports where > each port represents a different virtual function (port 1 is for the > uplink). All the limitations listed above are now resolved and the > software model exposed to users is more cohesive. > > Thanks > > Mark Bloch (12): > net/mlx5: E-Switch, don't use hardcoded values for FDB prios > net/mlx5: E-Switch, add a new prio to be used by the RDMA side > RDMA/mlx5: Move netdev info into the port struct > RDMA/mlx5: Free IB device on remove > RDMA/mlx5: Move ports allocation to outside of INIT stage > RDMA/mlx5: Use correct size for device resources > RDMA/mlx5: Move rep into port struct > RDMA/mlx5: Move default representors SQ steering to rule to modify QP > RDMA/mlx5: Refactor netdev affinity code > RDMA/mlx5: Move SMI caps logic > RDMA/mlx5: Move to single device multiport ports in switchdev mode > RDMA/mlx5: Remove VF representor profile This looks Ok, can you update the shared branch please Jason
On Thu, Apr 04, 2019 at 10:02:21AM -0300, Jason Gunthorpe wrote: > On Thu, Mar 28, 2019 at 03:27:30PM +0200, Leon Romanovsky wrote: > > From: Leon Romanovsky <leonro@mellanox.com> > > > > >From Mark, > > > > Hi, > > > > This series starts from two refactoring patches performed in mlx5_core > > and they are part of a larger series intended to expose the ability > > to insert FDB steering rules by the RDMA side. > > > > They are followed by patches which move IB representors to single IB > > device. Currently when moving to switchdev mode an IB device is created > > per virtual function (and one for the uplink). While initially this was > > right approach, the limitation of this model requires us to change it. > > > > Such limitations includes: > > - Inability to share resources between IB devices. > > - Lack of DEVX to allow more control and device specific functionality. > > - Interacting with the E-Switch. > > - Doesn't work well with namespaces. > > > > In this series we move to a single IB device with multiple ports where > > each port represents a different virtual function (port 1 is for the > > uplink). All the limitations listed above are now resolved and the > > software model exposed to users is more cohesive. > > > > Thanks > > > > Mark Bloch (12): > > net/mlx5: E-Switch, don't use hardcoded values for FDB prios > > net/mlx5: E-Switch, add a new prio to be used by the RDMA side > > RDMA/mlx5: Move netdev info into the port struct > > RDMA/mlx5: Free IB device on remove > > RDMA/mlx5: Move ports allocation to outside of INIT stage > > RDMA/mlx5: Use correct size for device resources > > RDMA/mlx5: Move rep into port struct > > RDMA/mlx5: Move default representors SQ steering to rule to modify QP > > RDMA/mlx5: Refactor netdev affinity code > > RDMA/mlx5: Move SMI caps logic > > RDMA/mlx5: Move to single device multiport ports in switchdev mode > > RDMA/mlx5: Remove VF representor profile > > This looks Ok, can you update the shared branch please Thanks Jason, I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch. Thanks > > Jason
On Thu, Apr 04, 2019 at 08:42:38PM +0300, Leon Romanovsky wrote: > On Thu, Apr 04, 2019 at 10:02:21AM -0300, Jason Gunthorpe wrote: > > On Thu, Mar 28, 2019 at 03:27:30PM +0200, Leon Romanovsky wrote: > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > >From Mark, > > > > > > Hi, > > > > > > This series starts from two refactoring patches performed in mlx5_core > > > and they are part of a larger series intended to expose the ability > > > to insert FDB steering rules by the RDMA side. > > > > > > They are followed by patches which move IB representors to single IB > > > device. Currently when moving to switchdev mode an IB device is created > > > per virtual function (and one for the uplink). While initially this was > > > right approach, the limitation of this model requires us to change it. > > > > > > Such limitations includes: > > > - Inability to share resources between IB devices. > > > - Lack of DEVX to allow more control and device specific functionality. > > > - Interacting with the E-Switch. > > > - Doesn't work well with namespaces. > > > > > > In this series we move to a single IB device with multiple ports where > > > each port represents a different virtual function (port 1 is for the > > > uplink). All the limitations listed above are now resolved and the > > > software model exposed to users is more cohesive. > > > > > > Thanks > > > > > > Mark Bloch (12): > > > net/mlx5: E-Switch, don't use hardcoded values for FDB prios > > > net/mlx5: E-Switch, add a new prio to be used by the RDMA side > > > RDMA/mlx5: Move netdev info into the port struct > > > RDMA/mlx5: Free IB device on remove > > > RDMA/mlx5: Move ports allocation to outside of INIT stage > > > RDMA/mlx5: Use correct size for device resources > > > RDMA/mlx5: Move rep into port struct > > > RDMA/mlx5: Move default representors SQ steering to rule to modify QP > > > RDMA/mlx5: Refactor netdev affinity code > > > RDMA/mlx5: Move SMI caps logic > > > RDMA/mlx5: Move to single device multiport ports in switchdev mode > > > RDMA/mlx5: Remove VF representor profile > > > > This looks Ok, can you update the shared branch please > > Thanks Jason, > > I'll slightly delay update till Sunday, internally, we have doubts about > "RDMA/mlx5: Use correct size for device resources" patch. Hi Jason, I updated our mlx5-next with first two patches. d9cb06759eca net/mlx5: E-Switch, add a new prio to be used by the RDMA side b6d9ccb11250 net/mlx5: E-Switch, don't use hardcoded values for FDB prios We found the reason for instability over bond interface and it was related to the patch "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. It will be helpful if you can apply it internally, or I can resend rest of the series. commit 775efa8dd53e5148d8d4942d2d0b7e72ca606d40 Author: Mark Bloch <markb@mellanox.com> Date: Tue Apr 9 03:28:59 2019 +0000 fixup! RDMA/mlx5: Move netdev info into the port struct diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index fdcc76383ae9..6f479dc66b78 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, print_lib_caps(dev, context->lib_caps); if (dev->lag_active) { - u8 port = mlx5_core_native_port_num(dev->mdev); + u8 port = mlx5_core_native_port_num(dev->mdev) - 1; atomic_set(&context->tx_port_affinity, atomic_add_return( diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 832f6d176e25..31039fe1dd20 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, (ibqp->qp_type == IB_QPT_XRC_INI) || (ibqp->qp_type == IB_QPT_XRC_TGT)) { if (dev->lag_active) { - u8 p = mlx5_core_native_port_num(dev->mdev); + u8 p = mlx5_core_native_port_num(dev->mdev) - 1; tx_affinity = get_tx_affinity(dev, pd, base, p, udata); context->flags |= cpu_to_be32(tx_affinity << 24); > > Thanks > > > > > Jason
On Wed, Apr 10, 2019 at 09:48:40AM +0300, Leon Romanovsky wrote: > > > > Mark Bloch (12): > > > > net/mlx5: E-Switch, don't use hardcoded values for FDB prios > > > > net/mlx5: E-Switch, add a new prio to be used by the RDMA side > > > > RDMA/mlx5: Move netdev info into the port struct > > > > RDMA/mlx5: Free IB device on remove > > > > RDMA/mlx5: Move ports allocation to outside of INIT stage > > > > RDMA/mlx5: Use correct size for device resources > > > > RDMA/mlx5: Move rep into port struct > > > > RDMA/mlx5: Move default representors SQ steering to rule to modify QP > > > > RDMA/mlx5: Refactor netdev affinity code > > > > RDMA/mlx5: Move SMI caps logic > > > > RDMA/mlx5: Move to single device multiport ports in switchdev mode > > > > RDMA/mlx5: Remove VF representor profile > > > > > > This looks Ok, can you update the shared branch please > > > > Thanks Jason, > > > > I'll slightly delay update till Sunday, internally, we have doubts about > > "RDMA/mlx5: Use correct size for device resources" patch. > > Hi Jason, > > I updated our mlx5-next with first two patches. > d9cb06759eca net/mlx5: E-Switch, add a new prio to be used by the RDMA side > b6d9ccb11250 net/mlx5: E-Switch, don't use hardcoded values for FDB prios > > We found the reason for instability over bond interface and it was related to the patch > "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. > It will be helpful if you can apply it internally, or I can resend rest of the series. > > commit 775efa8dd53e5148d8d4942d2d0b7e72ca606d40 > Author: Mark Bloch <markb@mellanox.com> > Date: Tue Apr 9 03:28:59 2019 +0000 > > fixup! RDMA/mlx5: Move netdev info into the port struct > > diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c > index fdcc76383ae9..6f479dc66b78 100644 > +++ b/drivers/infiniband/hw/mlx5/main.c > @@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, > print_lib_caps(dev, context->lib_caps); > > if (dev->lag_active) { > - u8 port = mlx5_core_native_port_num(dev->mdev); > + u8 port = mlx5_core_native_port_num(dev->mdev) - 1; > > atomic_set(&context->tx_port_affinity, > atomic_add_return( > diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c > index 832f6d176e25..31039fe1dd20 100644 > +++ b/drivers/infiniband/hw/mlx5/qp.c > @@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, > (ibqp->qp_type == IB_QPT_XRC_INI) || > (ibqp->qp_type == IB_QPT_XRC_TGT)) { > if (dev->lag_active) { > - u8 p = mlx5_core_native_port_num(dev->mdev); > + u8 p = mlx5_core_native_port_num(dev->mdev) - 1; > tx_affinity = get_tx_affinity(dev, pd, base, p, > udata); > context->flags |= cpu_to_be32(tx_affinity << 24); Okay, done Jason
From: Leon Romanovsky <leonro@mellanox.com> From Mark, Hi, This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side. They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it. Such limitations includes: - Inability to share resources between IB devices. - Lack of DEVX to allow more control and device specific functionality. - Interacting with the E-Switch. - Doesn't work well with namespaces. In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive. Thanks Mark Bloch (12): net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code RDMA/mlx5: Move SMI caps logic RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile drivers/infiniband/hw/mlx5/devx.c | 2 +- drivers/infiniband/hw/mlx5/flow.c | 2 +- drivers/infiniband/hw/mlx5/ib_rep.c | 109 +++++---- drivers/infiniband/hw/mlx5/ib_rep.h | 13 +- drivers/infiniband/hw/mlx5/main.c | 207 ++++++++++-------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 35 +-- drivers/infiniband/hw/mlx5/mr.c | 6 +- drivers/infiniband/hw/mlx5/qp.c | 46 ++-- .../mellanox/mlx5/core/eswitch_offloads.c | 5 - .../net/ethernet/mellanox/mlx5/core/fs_core.c | 12 +- include/linux/mlx5/fs.h | 6 + 11 files changed, 239 insertions(+), 204 deletions(-) -- 2.20.1