Message ID | 7094690ae0a8a43bc100f16dd4d4e6b9f1c0fc2a.1544597914.git-series.andrew.donnellan@au1.ibm.com |
---|---|
State | Superseded |
Headers | show |
Series | Support OpenCAPI and NVLink devices on same NPU on Witherspoon | expand |
Context | Check | Description |
---|---|---|
snowpatch_ozlabs/apply_patch | success | master/apply_patch Successfully applied |
snowpatch_ozlabs/snowpatch_job_snowpatch-skiboot | success | Test snowpatch/job/snowpatch-skiboot on branch master |
Le 12/12/2018 à 07:58, Andrew Donnellan a écrit : > When setting up an NPU with an OpenCAPI device, we need to mask the FIR > bits for NDL Stall/NoStall, which are used for a different purpose on > OpenCAPI vs NVLink. > > Currently, we just mask the bits for all links/DLs. When we support mixed > setups of OpenCAPI + NVLink on the same NPU, we don't want to mask all the > bits. > > Only mask the FIR bits for the specific links which are OpenCAPI. > > Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> > --- Funny how the code ends up being much simpler than what the documentation makes it look like :-) Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com> > hw/npu2-opencapi.c | 30 +++++++++++++++++++++--------- > 1 file changed, 21 insertions(+), 9 deletions(-) > > diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c > index 8ecd323416f4..d79727ad4199 100644 > --- a/hw/npu2-opencapi.c > +++ b/hw/npu2-opencapi.c > @@ -1303,7 +1303,8 @@ static void npu2_opencapi_final_fixup(struct phb *phb) > > static void mask_nvlink_fir(struct npu2 *p) > { > - uint64_t reg; > + uint64_t reg, mask = 0ull; > + int link_num; > > /* > * From section 13.1.3.10 of the NPU workbook: "the NV-Link > @@ -1312,35 +1313,46 @@ static void mask_nvlink_fir(struct npu2 *p) > * OpenCAPI. Therefore, the corresponding bits in NPU FIR > * Register 1 must be masked and configured to NOT cause the > * NPU to go into Freeze or Fence mode or send an Interrupt." > - * > - * FIXME: will need to revisit when mixing nvlink with > - * opencapi. Assumes an opencapi-only setup on both PHYs for > - * now. > */ > > + for (int i = 0; i < p-> total_devices; i++) { > + struct npu2_dev *dev = &p->devices[i]; > + /* Only mask OpenCAPI links */ > + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) > + continue; > + > + if (dev->brick_index == 2 || dev->brick_index == 3) { > + link_num = dev->brick_index - 2; > + } else { > + link_num = dev->brick_index; > + } > + mask = SETFIELD(PPC_BITMASK(link_num * 2, > + link_num * 2 + 1), > + mask, 0b11); > + } > /* Mask FIRs */ > xscom_read(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, ®); > - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0xFFF); > + reg |= mask; > xscom_write(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, reg); > > /* freeze disable */ > reg = npu2_scom_read(p->chip_id, p->xscom_base, > NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B); > - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); > + reg &= ~mask; > npu2_scom_write(p->chip_id, p->xscom_base, > NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); > > /* fence disable */ > reg = npu2_scom_read(p->chip_id, p->xscom_base, > NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B); > - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); > + reg &= ~mask; > npu2_scom_write(p->chip_id, p->xscom_base, > NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); > > /* irq disable */ > reg = npu2_scom_read(p->chip_id, p->xscom_base, > NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B); > - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); > + reg &= ~mask; > npu2_scom_write(p->chip_id, p->xscom_base, > NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); > } >
diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 8ecd323416f4..d79727ad4199 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1303,7 +1303,8 @@ static void npu2_opencapi_final_fixup(struct phb *phb) static void mask_nvlink_fir(struct npu2 *p) { - uint64_t reg; + uint64_t reg, mask = 0ull; + int link_num; /* * From section 13.1.3.10 of the NPU workbook: "the NV-Link @@ -1312,35 +1313,46 @@ static void mask_nvlink_fir(struct npu2 *p) * OpenCAPI. Therefore, the corresponding bits in NPU FIR * Register 1 must be masked and configured to NOT cause the * NPU to go into Freeze or Fence mode or send an Interrupt." - * - * FIXME: will need to revisit when mixing nvlink with - * opencapi. Assumes an opencapi-only setup on both PHYs for - * now. */ + for (int i = 0; i < p-> total_devices; i++) { + struct npu2_dev *dev = &p->devices[i]; + /* Only mask OpenCAPI links */ + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + + if (dev->brick_index == 2 || dev->brick_index == 3) { + link_num = dev->brick_index - 2; + } else { + link_num = dev->brick_index; + } + mask = SETFIELD(PPC_BITMASK(link_num * 2, + link_num * 2 + 1), + mask, 0b11); + } /* Mask FIRs */ xscom_read(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, ®); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0xFFF); + reg |= mask; xscom_write(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, reg); /* freeze disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* fence disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* irq disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); }
When setting up an NPU with an OpenCAPI device, we need to mask the FIR bits for NDL Stall/NoStall, which are used for a different purpose on OpenCAPI vs NVLink. Currently, we just mask the bits for all links/DLs. When we support mixed setups of OpenCAPI + NVLink on the same NPU, we don't want to mask all the bits. Only mask the FIR bits for the specific links which are OpenCAPI. Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> --- hw/npu2-opencapi.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)