| Message ID | 20180719075500.17822-2-vaibhav@linux.ibm.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [1/2] phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | master/apply_patch Successfully applied |
| snowpatch_ozlabs/make_check | success | Test make_check on branch master |
On 19/07/18 17:55, Vaibhav Jain wrote: > We add a man page describing the opal call OPAL_PCI_SET_PHB_CAPI_MODE > used for activating/deactivating CAPP attached to a PEC for CAPI 1 & 2. > > Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com> A few nitpicks below but they can perhaps be fixed in a followup if we want to merge this series now. Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> > --- > .../opal-pci-set-phb-capi-mode-93.rst | 74 +++++++++++++++++++ > 1 file changed, 74 insertions(+) > create mode 100644 doc/opal-api/opal-pci-set-phb-capi-mode-93.rst > > diff --git a/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst > new file mode 100644 > index 00000000..c0c7bd8d > --- /dev/null > +++ b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst > @@ -0,0 +1,74 @@ > +OPAL_PCI_SET_PHB_CAPI_MODE > +=========================== > + > +Switch the CAPP attached to the given PHB in one of the supported CAPI modes > + > +Parameters > +---------- > +``uint64_t phb_id`` > + the ID of the PHB which identifies attached CAPP to perform mode switch on > + > +``uint64_t mode`` > + A mode id as described below > + > +``pe_number`` > + PE number for the initiating device > + > +Calling > +------- > + > +Switch CAPP attached to the given PHB in one of the following supported modes: > + > +OPAL_PHB_CAPI_MODE_PCIE = 0 > +OPAL_PHB_CAPI_MODE_CAPI = 1 > +OPAL_PHB_CAPI_MODE_SNOOP_OFF = 2 > +OPAL_PHB_CAPI_MODE_SNOOP_ON = 3 > +OPAL_PHB_CAPI_MODE_DMA = 4 > +OPAL_PHB_CAPI_MODE_DMA_TVT1 = 5 > + > +Modes `OPAL_PHB_CAPI_MODE_PCIE` and `OPAL_PHB_CAPI_MODE_CAPI` are used to > +enable/disable CAPP attached to the PHB. > + > +Modes `OPAL_PHB_CAPI_MODE_SNOOP`_OFF and `OPAL_PHB_CAPI_MODE_SNOOP_ON` are The ` should come after _OFF > +used to enable/disable CAPP snooping of Powerbus traffic for cache line > +invalidates. > + > +Mode `OPAL_PHB_CAPI_MODE_DMA` and `OPAL_PHB_CAPI_MODE_DMA_TVT1` are used to > +enable CAPP DMA mode. > + > +Presently Mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` is exclusively used by the Mellanox > +CX5 adapter. Requesting this mode will also indicate to opal that the card > +requests maximum number of DMA read engines allocated to improve DMA read > +performance at cost of reduced bandwidth available to other traffic including > +CAPP-PSL transactions. > + > +Notes: > +----- > +* If PHB is in PEC2 then requesting mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` will > + allocate extra 16/8 dma read engines to the PHB depending on its stack > + (stack 0/ stack 1). This is needed to improve the Direct-GPU DMA read > + performance for the Mellanox CX5 card. > +* Mode `OPAL_PHB_CAPI_MODE_PCIE` not yet supported on Power-9. Or on POWER8. > +* Requesting mode `OPAL_PHB_CAPI_MODE_CAPI` on Power-9 will disable fast-reboot. > +* Modes `OPAL_PHB_CAPI_MODE_DMA`, `OPAL_PHB_CAPI_MODE_SNOOP_OFF` are > + not supported on Power-9 yet. > + > +Return Codes > +------------ > +OPAL_SUCCESS > + Switch to the reqeuested capi mode performed successfully. requested > + > +OPAL_PARAMETER > + The requested value of mode or phb_id parameter is not valid. > + > +OPAL_HARDWARE > + An error occurred while switching the CAPP to requested mode. > + > +OPAL_UNSUPPORTED > + Switching to requested capi mode is not possible at the moment > + > +OPAL_RESOURCE > + CAPP ucode not available hence activating CAPP not supported. > + > +OPAL_BUSY > + CAPP is presently in recovery-mode and mode switch cannot be performed. >
diff --git a/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst new file mode 100644 index 00000000..c0c7bd8d --- /dev/null +++ b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst @@ -0,0 +1,74 @@ +OPAL_PCI_SET_PHB_CAPI_MODE +=========================== + +Switch the CAPP attached to the given PHB in one of the supported CAPI modes + +Parameters +---------- +``uint64_t phb_id`` + the ID of the PHB which identifies attached CAPP to perform mode switch on + +``uint64_t mode`` + A mode id as described below + +``pe_number`` + PE number for the initiating device + +Calling +------- + +Switch CAPP attached to the given PHB in one of the following supported modes: + +OPAL_PHB_CAPI_MODE_PCIE = 0 +OPAL_PHB_CAPI_MODE_CAPI = 1 +OPAL_PHB_CAPI_MODE_SNOOP_OFF = 2 +OPAL_PHB_CAPI_MODE_SNOOP_ON = 3 +OPAL_PHB_CAPI_MODE_DMA = 4 +OPAL_PHB_CAPI_MODE_DMA_TVT1 = 5 + +Modes `OPAL_PHB_CAPI_MODE_PCIE` and `OPAL_PHB_CAPI_MODE_CAPI` are used to +enable/disable CAPP attached to the PHB. + +Modes `OPAL_PHB_CAPI_MODE_SNOOP`_OFF and `OPAL_PHB_CAPI_MODE_SNOOP_ON` are +used to enable/disable CAPP snooping of Powerbus traffic for cache line +invalidates. + +Mode `OPAL_PHB_CAPI_MODE_DMA` and `OPAL_PHB_CAPI_MODE_DMA_TVT1` are used to +enable CAPP DMA mode. + +Presently Mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` is exclusively used by the Mellanox +CX5 adapter. Requesting this mode will also indicate to opal that the card +requests maximum number of DMA read engines allocated to improve DMA read +performance at cost of reduced bandwidth available to other traffic including +CAPP-PSL transactions. + +Notes: +----- +* If PHB is in PEC2 then requesting mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` will + allocate extra 16/8 dma read engines to the PHB depending on its stack + (stack 0/ stack 1). This is needed to improve the Direct-GPU DMA read + performance for the Mellanox CX5 card. +* Mode `OPAL_PHB_CAPI_MODE_PCIE` not yet supported on Power-9. +* Requesting mode `OPAL_PHB_CAPI_MODE_CAPI` on Power-9 will disable fast-reboot. +* Modes `OPAL_PHB_CAPI_MODE_DMA`, `OPAL_PHB_CAPI_MODE_SNOOP_OFF` are + not supported on Power-9 yet. + +Return Codes +------------ +OPAL_SUCCESS + Switch to the reqeuested capi mode performed successfully. + +OPAL_PARAMETER + The requested value of mode or phb_id parameter is not valid. + +OPAL_HARDWARE + An error occurred while switching the CAPP to requested mode. + +OPAL_UNSUPPORTED + Switching to requested capi mode is not possible at the moment + +OPAL_RESOURCE + CAPP ucode not available hence activating CAPP not supported. + +OPAL_BUSY + CAPP is presently in recovery-mode and mode switch cannot be performed.
We add a man page describing the opal call OPAL_PCI_SET_PHB_CAPI_MODE used for activating/deactivating CAPP attached to a PEC for CAPI 1 & 2. Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com> --- .../opal-pci-set-phb-capi-mode-93.rst | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 doc/opal-api/opal-pci-set-phb-capi-mode-93.rst