From patchwork Thu Feb 27 20:40:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Grimm X-Patchwork-Id: 1246089 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48T4MP392tz9sQt for ; Fri, 28 Feb 2020 07:41:21 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 48T4MP24qHzDr6X for ; Fri, 28 Feb 2020 07:41:21 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=grimm@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 48T4LY6jXZzDr68 for ; Fri, 28 Feb 2020 07:40:37 +1100 (AEDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 01RKT0Wf141044 for ; Thu, 27 Feb 2020 15:40:35 -0500 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 2ye71287bw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 27 Feb 2020 15:40:34 -0500 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 01RKWplS180522 for ; Thu, 27 Feb 2020 15:40:34 -0500 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 2ye71287bp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 27 Feb 2020 15:40:34 -0500 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 01RKeEgo002950; Thu, 27 Feb 2020 20:40:33 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma04wdc.us.ibm.com with ESMTP id 2ydcmm0379-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 27 Feb 2020 20:40:33 +0000 Received: from b03ledav003.gho.boulder.ibm.com (b03ledav003.gho.boulder.ibm.com [9.17.130.234]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 01RKeVQN48431458 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 27 Feb 2020 20:40:31 GMT Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8AE126A047; Thu, 27 Feb 2020 20:40:31 +0000 (GMT) Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 890296A04D; Thu, 27 Feb 2020 20:40:30 +0000 (GMT) Received: from alain.ibm.com (unknown [9.80.218.175]) by b03ledav003.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 27 Feb 2020 20:40:30 +0000 (GMT) From: Ryan Grimm To: oohall@gmail.com Date: Thu, 27 Feb 2020 15:40:08 -0500 Message-Id: <20200227204023.22125-2-grimm@linux.ibm.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20200227204023.22125-1-grimm@linux.ibm.com> References: <20200227204023.22125-1-grimm@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-02-27_07:2020-02-26, 2020-02-27 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 mlxscore=0 bulkscore=0 suspectscore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002270140 Subject: [Skiboot] [RFC PATCH v5 01/16] doc/opal-uv-abi.rst X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janani@us.ibm.com, suka@us.ibm.com, skiboot@lists.ozlabs.org Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Signed-off-by: Ryan Grimm --- doc/opal-uv-abi.rst | 423 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 423 insertions(+) create mode 100644 doc/opal-uv-abi.rst diff --git a/doc/opal-uv-abi.rst b/doc/opal-uv-abi.rst new file mode 100644 index 00000000..bf91e730 --- /dev/null +++ b/doc/opal-uv-abi.rst @@ -0,0 +1,423 @@ +.. SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-or-later + +================= +OPAL UV ABI (RFC) +================= + +.. contents:: + :depth: 3 + +.. sectnum:: + :depth: 3 + +This document describes the function calling interface between OPAL +and the Ultravisor. + +Protected Execution Facility +############################ + +Protected Execution Facility (PEF) is an architectural change for +POWER 9 that enables Secure Virtual Machines (SVMs). When enabled, +PEF adds a new higher privileged mode, called Ultravisor mode, to +POWER architecture. Along with the new mode there is new firmware +called the Protected Execution Ultravisor (or Ultravisor for short). +Ultravisor mode is the highest privileged mode in POWER architecture. + ++------------------+ +| Privilege States | ++==================+ +| Problem | ++------------------+ +| Supervisor | ++------------------+ +| Hypervisor | ++------------------+ +| Ultravisor | ++------------------+ + +PEF protects SVMs from the hypervisor, privileged users, and other +VMs in the system. SVMs are protected while at rest and can only be +executed by an authorized machine. All virtual machines utilize +hypervisor services. The Ultravisor filters calls between the SVMs +and the hypervisor to assure that information does not accidentally +leak. All hypercalls except H_RANDOM are reflected to the hypervisor. +H_RANDOM is not reflected to prevent the hypervisor from influencing +random values in the SVM. + +To support this there is a refactoring of the ownership of resources +in the CPU. Some of the resources which were previously hypervisor +privileged are now ultravisor privileged. + +Hardware +======== + +The hardware changes include the following: + +* There is a new bit in the MSR that determines whether the current + process is running in secure mode, MSR(S) bit 41. MSR(S)=1, process + is in secure mode, MSR(s)=0 process is in normal mode. + +* The MSR(S) bit can only be set by the Ultravisor. + +* HRFID cannot be used to set the MSR(S) bit. If the hypervisor needs + to return to a SVM it must use an ultracall. It can determine if + the VM it is returning to is secure. + +* There is a new Ultravisor privileged register, SMFCTRL, which has an + enable/disable bit SMFCTRL(E). + +* The privilege of a process is now determined by three MSR bits, + MSR(S, HV, PR). In each of the tables below the modes are listed + from least privilege to highest privilege. The higher privilege + modes can access all the resources of the lower privilege modes. + +**Secure Mode MSR Settings** + ++---+---+---+---------------+ +| S | HV| PR|Privilege | ++===+===+===+===============+ +| 1 | 0 | 1 | Problem | ++---+---+---+---------------+ +| 1 | 0 | 0 | Privileged(OS)| ++---+---+---+---------------+ +| 1 | 1 | 0 | Ultravisor | ++---+---+---+---------------+ +| 1 | 1 | 1 | Reserved | ++---+---+---+---------------+ + +**Normal Mode MSR Settings** + ++---+---+---+---------------+ +| S | HV| PR|Privilege | ++===+===+===+===============+ +| 0 | 0 | 1 | Problem | ++---+---+---+---------------+ +| 0 | 0 | 0 | Privileged(OS)| ++---+---+---+---------------+ +| 0 | 1 | 0 | Hypervisor | ++---+---+---+---------------+ +| 0 | 1 | 1 | Problem (HV) | ++---+---+---+---------------+ + +* Memory is partitioned into secure and normal memory. Only processes + that are running in secure mode can access secure memory. + +* The hardware does not allow anything that is not running secure to + access secure memory. This means that the Hypervisor cannot access + the memory of the SVM without using an ultracall (asking the + Ultravisor). The Ultravisor will only allow the hypervisor to see + the SVM memory encrypted. + +* I/O systems are not allowed to directly address secure memory. This + limits the SVMs to virtual I/O only. + +* The architecture allows the SVM to share pages of memory with the + hypervisor that are not protected with encryption. However, this + sharing must be initiated by the SVM. + +* When a process is running in secure mode all hypercalls + (syscall lev=1) are reflected to the Ultravisor. + +* When a process is in secure mode all interrupts go to the + Ultravisor. + +* The following resources have become Ultravisor privileged and + require an Ultravisor interface to manipulate: + + * Processor configurations registers (SCOMs). + + * Stop state information. + + * The debug registers CIABR, DAWR, and DAWRX become Ultravisor + resources when SMFCTRL(D) is set. If SMFCTRL(D) is not set they do + not work in secure mode. When set, reading and writing requires + an Ultravisor call, otherwise that will cause a Hypervisor Emulation + Assistance interrupt. + + * PTCR and partition table entries (partition table is in secure + memory). An attempt to write to PTCR will cause a Hypervisor + Emulation Assitance interrupt. + + * LDBAR (LD Base Address Register) and IMC (In-Memory Collection) + non-architected registers. An attempt to write to them will cause a + Hypervisor Emulation Assistance interrupt. + + * Paging for an SVM, sharing of memory with Hypervisor for an SVM. + (Including Virtual Processor Area (VPA) and virtual I/O). + +Software/Microcode +================== + +The software changes include: + +* When the UV_ESM ultracall is made the Ultravisor copies the VM into + secure memory, decrypts the verification information, and checks the + integrity of the SVM. If the integrity check passes the Ultravisor + passes control in secure mode. + +The Ultravisor offers new services to the hypervisor and SVMs. These +are accessed through ultracalls. + +Terminology +=========== + +* Hypercalls: special system calls used to request services from + Hypervisor. + +* Normal memory: Memory that is accessible to Hypervisor. + +* Normal page: Page backed by normal memory and available to + Hypervisor. + +* Secure memory: Memory that is accessible only to Ultravisor and + SVMs. + +* Secure page: Page backed by secure memory and only available to + Ultravisor and SVM. + +* SVM: Secure Virtual Machine. + +* Ultracalls: special system calls used to request services from + Ultravisor. + +Ultravisor Initialization +######################### + +Secure Memory +============= + +Skiboot parses secure memory from the HDAT tables and creates the secure-memory +and ibm,ultravisor device tree nodes. secure-memory similar to a memory@ node +except the device_type is "secure_memory". For example: + +.. code-block:: dts + + secure-memory@100fe00000000 { + device_type = "secure_memory"; + compatible = "ibm,secure_memory"; + ibm,chip-id = <0>; + reg = < 0x100fe 0x0 0x2 0x0>; + } + +Regions of secure memory will be reserved by hostboot such as OCC, HOMER, and +SBE. Skiboot will use the existing reserve infrastructure to reserve them. +For example: + +.. code-block:: + + ibm,HCODE@100fffcaf0000 + ibm,OCC@100fffcdd0000 + ibm,RINGOVD@100fffcae0000 + ibm,WOFDATA@100fffcb90000 + ibm,arch-reg-data@100fffd700000 + ibm,hbrt-code-image@100fffcec0000 + ibm,hbrt-data@100fffd420000 + ibm,homer-image@100fffd800000 + ibm,homer-image@100fffdc00000 + ibm,occ-common-area@100ffff800000 + ibm,sbe-comm@100fffce90000 + ibm,sbe-comm@100fffceb0000 + ibm,sbe-ffdc@100fffce80000 + ibm,sbe-ffdc@100fffcea0000 + ibm,secure-crypt-algo-code@100fffce70000 + ibm,uvbwlist@100fffcad0000 + +For Mambo, ultra.tcl creates the secure-memory device tree node and the +ibm,ultravisor device tree node in external/mambo/skiboot.tcl. Secure memory +is currently defined as the bottom half of the total the size of memory. Mambo +has no protection on secure memory, so a watchpoint could be used to ensure +Skiboot does not touch secure memory. + +For BML, the BML script parses secure memory from the Cronus config file and +creates the secure-memory and ibm,ultravisor device tree nodes. + +In all cases, the console log should indicate secure memory has been found and +added to the device tree. For example: + +.. code-block:: + + [ 68.235326307,5] UV: Secure memory range added to DT [0x0001000e00000000..0x001001000000000] + +Loading The Ultravisor +====================== + +Skiboot uses secure and trusted boot to load and verify the compressed UV image +from the PNOR into regular memory. It unpacks the UV into regular memory in +the function ``init_uv``. + +``init_uv`` finds the UV node in the device tree via the "ibm,ultravisor" +compatible property. For example: + +.. code-block:: dts + + ibm,ultravisor { + compatible = "ibm,ultravisor"; + #address-cells = <0x02>; + #size-cells = <0x02>; + + firmware@200000000 { + compatible = "ibm,uv-firmware"; + reg = <0x02 0x00 0xf677f>; + memcons = <0x00 0x3022d030>; + sys-fdt = <0x00 0x30509068>; + uv-fdt = <0x02 0x200000>; + }; + }; + +Skiboot creates ibm,ultravisor and the reg property in hdata/spira.c. + +Mambo and BML use scripts to put the ultra image directly in regular memory and +a reserve is created named ibm,uv-firmware. + +Starting The Ultravisor +======================= + +Skiboot starts the UV in ``main_cpu_entry`` before the kernel is loaded and booted. +Skiboot creates a job on all threads and sends them to ``start_uv`` in asm/head.S. +This function's prototype is: + +.. code-block:: c + + /** + * @brief Start UV. + * + * @param uv_load_addr Load address of ultravisor. + * @param sys_fdt Pointer to system fdt + * + * @return 0 on success, else a negative error code on failure. + */ + u64 start_uv(u64 uv_load_addr, void *sys_fdt) + +The sys_fdt allows passing information to the UV, such as the location of the +memory consolde ultravisor fdt and is easy to extend. + +The ``uv_fdt`` is constructed in secure memory. It is allocated after the +ultravisor image at ``uv_base_addr + UV_LOAD_MAX_SIZE``. This allows the +ultravisor to load at start of the first secure memory range and recover the +memory allocated to ``uv_fdt``. + +.. code-block:: dts + + ibm,uv-fdt { + compatible = "ibm,uv-fdt"; + wrapping-key-password = "gUMShz6l2x4O9IeHrvBSuBR0FYANZTYK"; + }; + +The UV parses ``sys_fdt``, creates internal structures, and threads return in +hypervisor privilege moded. + +If successful, skiboot sets a variable named ``uv_present`` to true. Skiboot +uses this variable to dermine if the UV is initialized and ready to perform +ucalls. + +uv_base_addr is used by skiboot to know where to jump to. + +Ultravisor Failed Start Recovery +================================ + +If the ultravisor fails to start it will return a error code to init_uv. +init_uv will print error messages to the skiboot log and attempt to free +structures associated with the ultravisor. + +Skiboot will continue to be in ultravisor privilege mode, and will need to +perform a recovery action. + +[**TODO**: Need to describe the steps for Ultravisor load failure recovery action.] + +Ultracalls +########## + +Ultravisor calls ABI +==================== + +This section describes Ultravisor calls (ultracalls) needed by skiboot. +The ultracalls allow the skiboot to request services from the +Ultravisor such as initializing a chip unit via XSCOM. + +The specific service needed from an ultracall is specified in register +R3 (the first parameter to the ultracall). Other parameters to the +ultracall, if any, are specified in registers R4 through R12. + +Return value of all ultracalls is in register R3. Other output values +from the ultracall, if any, are returned in registers R4 through R12. + +Each ultracall returns specific error codes, applicable in the context +of the ultracall. However, like with the PowerPC Architecture Platform +Reference (PAPR), if no specific error code is defined for a +particular situation, then the ultracall will fallback to an erroneous +parameter-position based code. i.e U_PARAMETER, U_P2, U_P3 etc +depending on the ultracall parameter that may have caused the error. + +For now this only covers ultracalls currently implemented and being used by +skiboot but others can be added here when it makes sense. + +The full specification for all ultracalls will eventually be made available in +the public/OpenPower version of the PAPR specification. + +Ultracalls used by Skiboot +========================== + +UV_READ_SCOM +------------ + +Perform an XSCOM read and put the value in a buffer. + +Syntax +~~~~~~ + +.. code-block:: c + + long ucall(unsigned long UV_READ_SCOM, + unsigned long *retbuf, + u64 partid, + u64 pcb_addr) + +Return values +~~~~~~~~~~~~~ + +* U_SUCCESS on success. +* U_PERMISSION if called from VM context. +* U_PARAMETER if invalid partiton or address. +* U_BUSY if unit is busy, need to retry. +* U_XSCOM_CHIPLET_OFF if cpu is asleep. +* U_XSCOM_PARTIAL_GOOD if partial good. +* U_XSCOM_ADDR_ERROR if address error. +* U_XSCOM_CLOCK_ERROR if clock error. +* U_XSCOM_PARITY_ERROR if parity error. +* U_XSCOM_TIMEOUT if timeout. + +UV_WRITE_SCOM +------------- + +Perform an XSCOM write. + +Syntax +~~~~~~ + +.. code-block:: c + + long ucall(unsigned long UV_WRITE_SCOM, + u64 partid, + u64 pcb_addr, + u64 val) + +Return values +~~~~~~~~~~~~~ + +One of the following values: + +* U_SUCCESS on success. +* U_PERMISSION if called from VM context. +* U_PARAMETER if invalid partiton. +* U_BUSY if unit is busy, need to retry. +* U_XSCOM_CHIPLET_OFF if cpu is asleep. +* U_XSCOM_PARTIAL_GOOD if partial good. +* U_XSCOM_ADDR_ERROR if address error. +* U_XSCOM_CLOCK_ERROR if clock error. +* U_XSCOM_PARITY_ERROR if parity error. +* U_XSCOM_TIMEOUT if timeout. + +References +########## + +.. [1] `Supporting Protected Computing on IBM Power Architecture `_