[2/2] doc: Start writing some in-tree documentation

diff --git a/doc/Makefile b/doc/Makefile
new file mode 100644
index 00000000..298ea9e2
--- /dev/null
+++ b/doc/Makefile
@@ -0,0 +1,19 @@ 
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
diff --git a/doc/components.rst b/doc/components.rst
new file mode 100644
index 00000000..fed52a1d
--- /dev/null
+++ b/doc/components.rst
@@ -0,0 +1,29 @@ 
+Petitboot Components
+====================
+
+Server
+------
+
+The core of Petitboot is the ``pb-discover`` process. This performs initial setup, discovers and configures devices, and handles configuration options or quirks saved or set by the platform.
+
+UI Clients
+----------
+
+The ``ui/`` directory contains client processes that implement a user interface. The primary interface is ``petitboot-nc`` which is based on ncurses and provides the full range of options for interaction and configuration.
+
+There is also a twin-based interface under ``ui/twin/`` however this is largely a remnant from the PS3 and does not implement more recent functionality.
+
+Clients generally take no direct action themselves, instead communcating via the "pb-protocol" interface to the ``pb-discover`` server to request actions.
+
+Utilities
+---------
+
+A number of smaller utilities exist to perform some specific tasks, including:
+
+pb-console: Initial console setup and UI startup
+pb-config: Trimmed down 'client' that can request information from the ``pb-discover`` server.
+pb-event: Provides a callable script to send user events to ``pb-discover`` - usually used by other utilities such as ``udhcpc`` to report network information.
+pb-exec: Simple wrapper for running programs from the UI
+pb-plugin: Implements the petitboot-plugin interface
+pb-sos: Collects diagnostic and crash information
+boot hooks: Small hooks to be run immediately before boot.
diff --git a/doc/conf.py b/doc/conf.py
new file mode 100644
index 00000000..1aca405c
--- /dev/null
+++ b/doc/conf.py
@@ -0,0 +1,55 @@ 
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'Petitboot'
+copyright = '2019, Samuel Mendoza-Jonas, IBM'
+author = 'Samuel Mendoza-Jonas'
+
+# The full version, including alpha/beta/rc tags
+release = 'v1.10.3'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
diff --git a/doc/dev/design.rst b/doc/dev/design.rst
new file mode 100644
index 00000000..7c3fa220
--- /dev/null
+++ b/doc/dev/design.rst
@@ -0,0 +1,31 @@ 
+Design Topics
+=============
+
+When considering new functionality or other changes to Petitboot there are a few guidelines we do our best to adhere to:
+
+Be Generic
+----------
+
+Avoid adding code that ties Petitboot to a particular platform or functionality by default. Avoid making assumptions about the system that Petitboot may be running on. Platform specific functionality should as much as possible be confined to `platform-` files. Especially when thinking about dependencies on outside utilities or tools consider using the pb-plugin interface instead.
+
+Don't Surprise The User
+-----------------------
+
+Communicate clearly to the user what is happening. What the UI says is happening should be what actually happens.
+A particular example of this is the kernel command line: there are a few problems that would be easier to solve if we could modify the command line of the kernel to be booted, but this would be something that interferes with the users expectations and depending on their scenario could cause them problems.
+
+Avoid Dependencies On The Target Operating System
+-------------------------------------------------
+
+Supporting multiple configuration formats means that Petitboot can be dropped in as the bootloader for most existing systems without any change in the target operating system or its bootloader configuration. Avoid any changes that would require a corresponding update in the target operating system or dependencies on any particular version of Petitboot.
+
+Maintain Server-Client Separation
+---------------------------------
+
+As much as possible the server should perform any actions, and the clients should only makes requests to the server. This is particularly true with the introduction of Petitboot "users" and password restrictions.
+An exception to this is the running of Petitboot plugins since these need to be run in the visible shell; however these are still subject to user permissions if enabled.
+
+Abstract Dependencies
+---------------------
+
+Avoid direct dependencies on certain utilities or versions of utilities. If there are several versions or interfaces to a utility that Petitboot might access, abstract these differences where possible. (See handling of different utilities such as tftp, udhcpc, etc).
diff --git a/doc/dev/linux-image.rst b/doc/dev/linux-image.rst
new file mode 100644
index 00000000..ba2b65fe
--- /dev/null
+++ b/doc/dev/linux-image.rst
@@ -0,0 +1,35 @@ 
+Linux Image Requirements
+========================
+
+Petitboot has a few requirements for the linux image it is built in to.
+For an in-practice example of how to build a Petitboot image see op-build_.
+
+For build-time dependencies see configure.ac_.
+
+.. _op-build: https://github.com/open-power/op-build/tree/master/openpower/package/petitboot
+.. _configure.ac: https://github.com/open-power/petitboot/blob/master/configure.ac
+
+Core dependencies
+-----------------
+
+* pb-discover expects to be run as root, or at least have permission for device management, executing kexec, etc.
+* udev: pb-discover discovers devices via libudev enumeration so a udev implementation must be present.
+  Following this any udev rules required for certain device types must also be present. Eg. op-build's inclusions_.
+* network utilities: pb-discover expects to have ``udhcpc`` available for DHCP, or a call-equivalent version. Similarly it expects ``tftp`` and ``wget`` binaries in order to download boot resources.
+* kexec: A kexec binary must be available. This is commonly kexec-lite_ however kexec-tools should also work.
+* LVM: Petitboot depends on libdevmapper, and also requires ``vgscan`` and ``vgchange`` to be available if order to setup logical volumes.
+
+.. _inclusions: https://github.com/open-power/op-build/blob/master/openpower/package/petitboot/63-md-raid-arrays.rules
+.. _kexec-lite: https://github.com/open-power/kexec-lite
+
+Optional dependencies
+---------------------
+
+* ``mdadm`` for software RAID handling.
+* libflash: On OPAL platforms Petitboot will use libflash to load firmware version information.
+* ipmi: If ``/dev/ipmi`` is available Petitboot will use it to source information from the BMC.
+* console setup: unless you have other requirements you probably want to be starting the UI through ``pb-console`` in which case it is useful to have this called by udev.
+  For example: petitboot-console-ui.rules_, or depending on if you're using ``agetty -l`` to log in a user, shell_profile_.
+
+.. _petitboot-console-ui.rules: https://github.com/open-power/op-build/blob/master/openpower/package/petitboot/petitboot-console-ui.rules
+.. _shell_profile: https://github.com/open-power/op-build/blob/master/openpower/package/petitboot/shell_profile
diff --git a/doc/dev/submitting.rst b/doc/dev/submitting.rst
new file mode 100644
index 00000000..fb968cda
--- /dev/null
+++ b/doc/dev/submitting.rst
@@ -0,0 +1,26 @@ 
+Development and Submitting Patches
+==================================
+
+Petitboot is largely written in C and follows the `Linux kernel coding style <https://github.com/torvalds/linux/blob/master/Documentation/process/coding-style.rst>`_.
+
+Development occurs on the `Petitboot mailing list <https://lists.ozlabs.org/listinfo/petitboot>`_.
+
+Petitboot also has a `Patchwork instance <http://patchwork.ozlabs.org/project/petitboot/list/>`_ that watches the list.
+
+Patch Guidelines
+----------------
+
+Patches should be sent to the mailing list generally following what you would see in `submitting-patches <https://github.com/torvalds/linux/blob/master/Documentation/process/submitting-patches.rst>`_
+
+In general if you generate the patch with ``git format-patch`` or ``git send-email`` you should be fine.
+
+Patches should have an obvious title and where necessary a clear commit message describing the changes.
+Avoid lumping unrelated changes together, instead putting them in separate patches in a logical order.
+If a patch is generally contained to one area (and it should be), it should generally be prefixed with the path of what it is changing, for example "discover/grub:" or "ui/ncurses:".
+
+If sending a new revision of a patch update the title to mention the verson (hint: ``git format-patch -v 2``) and include a short changelog under the ``---`` describing what changed between versions. Check out the list archives for `some examples <https://lists.ozlabs.org/pipermail/petitboot/2018-November/001188.html>`_.
+
+Stable Patches
+--------------
+
+Patches or upstream commits that need to be applied to a stable branch should be restricted to small, self-contained fixes as much as possible. Avoid backporting new features or invasive changes.
diff --git a/doc/func/autoboot.rst b/doc/func/autoboot.rst
new file mode 100644
index 00000000..77243910
--- /dev/null
+++ b/doc/func/autoboot.rst
@@ -0,0 +1,32 @@ 
+.. _auto-boot:
+
+Auto Boot Order
+===============
+
+With autoboot enabled Petitboot will consider the relative priority of each new default boot option it discovers to determine what should be booted automatically.
+
+Note that a boot option must be marked for autoboot ("default") in its own configuration file for Petitboot to autoboot it. For example:
+
+.. code-block::
+
+   default linux
+
+   label linux
+   kernel ./pxe/de-ad-de-ad-be-ef.vmlinuz
+   append command line initrd=./pxe/de-ad-de-ad-be-ef.initrd
+
+Boot Priority
+-------------
+
+Petitboot can prioritise boot options based on the following attributes:
+
+* Device type (Disk, Network, USB..)
+* Partition (eg, sda2)
+* Interface name (eg. eth0)
+* LVM logical-volume name (eg. "rhel7-boot-lv")
+
+Once started Petitboot will begin to countdown from the configured timeout (default 10 seconds). During this time any default boot option that is discovered is compared against the configured priority list and the current option with the highest priority. If the new option has a higher priority based on the boot order than the current option then it becomes the current option and will be booted once the countdown completes.
+
+Note that unlike some other bootloaders Petitboot does *not* wait for devices in the boot order. For example if the boot order was "Network, Disk" with a 10 second default and a disk option was found but a hypothetic network option would take longer than 10 seconds to be found (eg. slow network or DHCP server), then Petitboot won't know about it and will boot the disk option. In most cases the appropriate solution if a user runs into this is to increase the timeout value to a suitable length of time for their environment.
+
+Note that :ref:`ipmi` overrides will take precedence over any configured boot order.
diff --git a/doc/func/ipmi.rst b/doc/func/ipmi.rst
new file mode 100644
index 00000000..8f9fa053
--- /dev/null
+++ b/doc/func/ipmi.rst
@@ -0,0 +1,29 @@ 
+.. _ipmi:
+
+IPMI
+====
+
+Petitboot uses inband-IPMI_ on platforms that support it to report information or modify booting behaviour:
+
+.. _inband-IPMI: https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface
+
+Platform Information
+--------------------
+
+Basic BMC information is read via the "Get Device ID" and "Get LAN Configuration Parameters" commands and shown in the System Information screen. On some platforms with an AMI BMC the OEM command "Get Device ID Golden" will also be issued to get the BMC's secondary side information.
+
+Just before successful boot Petitboot will set the OS boot sensor (command `0x30`) to notify the BMC of a successful boot.
+
+Boot Device Overrides
+---------------------
+
+Petitboot supports IPMI boot device overrides. These can be set on the BMC to temporarily override the autoboot behaviour. These are limited to the broad types of "Network", "Disk", "Safe", "CDROM", and "Setup".
+
+If a boot option matches one of these options it receives the highest priority regardless of the boot order. The exceptions are "Setup" which temporarily disables autoboot, and "Safe" which does the same and also enters "safe mode" which prevents setting up any devices until the user issues a "Rescan" command.
+
+Boot Initiator Mailbox
+----------------------
+
+Petitboot also supports the "Boot Initiator Mailbox" parameter which behaves similarly to an override but lets a full replacement boot order be set. Support for this on the BMC side is relatively limited so far, but more details can be found here_.
+
+.. _here: https://sthbrx.github.io/blog/2018/12/19/ipmi-initiating-better-overrides/
diff --git a/doc/func/parsers.rst b/doc/func/parsers.rst
new file mode 100644
index 00000000..92281bd2
--- /dev/null
+++ b/doc/func/parsers.rst
@@ -0,0 +1,18 @@ 
+Configuration Parsers
+=====================
+
+Petitboot can read a variety of configuration formats. Generally it does this in one of two ways:
+
+Bison/Flex Parsers
+------------------
+
+The "grub2" and "native" parsers are implemented with the Bison parser and Flex lexer combo to define a grammar describing the configuration format. These parsers can hook into other Petitboot or C code as needed but for most scenarios just need a way to resolve resources (resource.h) and create boot options (device-handler.h).
+
+Writing a parser/grammar this way can be a bit more difficult at first but does lend itself to a more robust solution in the long term.
+
+Process-Pair Parsers
+--------------------
+
+Other parsers such as for PXE and SYSLINUX use a simpler mechanism provided by `parser-conf.c` In short this provides a way to load the configuration into a buffer and process each line as a key:value format. Where applicable the parser can provide it's own callbacks for these functions.
+
+This method is a lot easier to quickly construct a parser, especially for formats with relatively basic formats.
diff --git a/doc/func/plugins.rst b/doc/func/plugins.rst
new file mode 100644
index 00000000..5cc471c8
--- /dev/null
+++ b/doc/func/plugins.rst
@@ -0,0 +1,27 @@ 
+.. _plugins:
+
+Plugins
+=======
+
+Petitboot "plugins" provide a convenient way to package and run binaries in the Petitboot shell that would be difficult to distribute in the Linux image due to size, dependency, or licensing constraints.
+
+Plugin usage and the plugin ABI are well documented in the OpenPOWER docs repository here:
+
+`Plugin Usage & Construction <https://github.com/open-power/docs/blob/master/opal/petitboot-plugins.txt>`_
+
+`Plugin Specification & ABI <https://github.com/open-power/docs/blob/master/opal/petitboot-plugin-spec.txt>`_
+
+Petitboot will scan local devices for pb-plugin files, and will also recognise the "plugin" label in PXE network files, eg:
+
+.. code-block::
+
+   label Install Ubuntu 18.04
+	kernel tftp://server/ubuntu-18.04-ppc64el/vmlinux
+	initrd tftp://server/ubuntu-18.04-ppc64el/initrd.gz
+	append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false
+
+   plugin RAID Setup
+        tarball http://server/path/to/plugin-raid-1.0.pb-plugin
+
+Discovered plugins will be listed in the Plugin screen and can be installed and run from there, or run manually from the shell.
+
diff --git a/doc/func/snapshots.rst b/doc/func/snapshots.rst
new file mode 100644
index 00000000..799ff1da
--- /dev/null
+++ b/doc/func/snapshots.rst
@@ -0,0 +1,34 @@ 
+.. _snapshots:
+
+Snapshots
+=========
+
+By default Petitboot does not directly mount any block devices. Instead it uses the device-mapper snapshot_ device to mount an in-memory representation of the device. Any writes Petitboot or another part of the system may make to the device are written to memory rather than the physical device, providing a "real" read-only guarantee beyond just that provided by the filesystem.
+
+In normal operation this is completely transparent but there are two scenarios where actual writes are desired:
+
+.. _snapshot: https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
+
+Saving bootloader data
+----------------------
+
+Some configuration formats include directives to save some data before boot - like GRUB's ability to record the last option booted. Snapshots will prevent this but Petitboot can be configured to merge these writes back to the source device by enabling the ""Allow bootloader scripts to modify disks" option in the System Configuration screen.
+
+Manual interaction
+------------------
+
+Any writes to a block device first mounted by Petitboot will by default be thrown away. If a user makes changes to a disk or USB device for example that they want to preserve they can tell Petitboot to merge these writes to the source device with the "sync" event. For example if the user had written something to the sda2 partition, in the shell they can run:
+
+.. code-block::
+
+   pb-event sync@sda2
+
+Petitboot will handle the merging itself and remount the device read-only.
+
+If desired snapshots can also be disabled via the "petitboot,snapshots?" parameter. For example:
+
+.. code-block::
+
+   nvram --update-config petitboot,snapshots?=false
+
+As of the next boot Petitboot will mount block devices directly.
diff --git a/doc/func/user_interface.rst b/doc/func/user_interface.rst
new file mode 100644
index 00000000..f528e8a6
--- /dev/null
+++ b/doc/func/user_interface.rst
@@ -0,0 +1,51 @@ 
+User Interface
+==============
+
+There are two UI implementations but in practice the ncurses client `petitboot-nc` is the primary interface.
+
+The default view includes a list of discovered boot options and links to other sub screens.
+Common shortcuts include:
+
+======== =====================================
+Shortcut
+======== =====================================
+i        Open the System Information screen
+e        View/Edit a boot option details
+g        View the status update log
+l        Open the Language screen
+c        Open the System Configuration screen
+n        Create/Edit a boot option
+h        Show help text for the current screen
+x        Exit to the shell
+Ctrl-L   Refresh/Redraw the display
+======== =====================================
+
+System Information
+------------------
+
+Provides an overview of the system, in particular discovered partitions and network interfaces. Depending on the platform can also include information on firmware versions, BMC information, etc.
+
+System Configuration
+--------------------
+
+The primary method to configure Petitboot. From here the user can configure the :ref:`auto-boot` order, network interface behaviour, :ref:`snapshots` settings, and handle platform-specific options like :ref:`ipmi` overrides and default consoles.
+
+Boot Option Editor
+------------------
+
+The boot option editor displays the source device and boot resources for a given option. All fields are editable; paths are relative to the source device or full URLs for remote resources.
+
+Status Log
+----------
+
+The status log keeps a running log of all status updates that normally appear in the bottom line of the interface.
+
+Plugin Menu
+-----------
+
+The plugin menu displays a list of all known :ref:`plugins` and whether they are installed or not. "Enter" installs a plugin, and "e" on an installed interface enters a detailed view of the plugin from which a user can run the plugin.
+
+Shell
+-----
+
+At any time the user may drop to a shell by exiting the UI, depending on how Petitboot has been built. This is usually a relatively feature-ful ``sh`` shell with Busybox utilities but Petitboot doesn't make any particular guarantees about available tools aside from those it uses itself.
diff --git a/doc/index.rst b/doc/index.rst
new file mode 100644
index 00000000..fe956994
--- /dev/null
+++ b/doc/index.rst
@@ -0,0 +1,53 @@ 
+.. Petitboot documentation master file, created by
+   sphinx-quickstart on Fri May 31 11:49:47 2019.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Petitboot Documentation
+=====================================
+
+Overview
+========
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   components
+   platforms
+
+Development
+===========
+
+.. toctree::
+   :maxdepth: 1
+
+   dev/design
+   dev/submitting
+   dev/linux-image
+
+Functionality
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   func/parsers
+   func/user_interface
+   func/autoboot
+   func/snapshots
+   func/ipmi
+   func/plugins
+
+.. toctree::
+   :maxdepth: 2
+
+
+.. only:: latex
+
+   Indices and tables
+   ==================
+
+   * :ref:`genindex`
+   * :ref:`modindex`
+   * :ref:`search`
diff --git a/doc/overview.rst b/doc/overview.rst
new file mode 100644
index 00000000..57b64594
--- /dev/null
+++ b/doc/overview.rst
@@ -0,0 +1,12 @@ 
+Petitboot Overview
+==================
+
+Petitboot is a set of userspace programs to implement a kexec-based bootloader in a small Linux image. Broadly speaking Petitboot is split into a 'server' half which manages devices and finds bootable images, and a 'client' half which provides a user interface.
+
+Petitboot uses udev to discover block devices and network interfaces and parses a set of known locations for bootloader configurations. Petitboot supports a number of configuration formats including GRUB, SYSLINUX, and Yaboot. Booting is performed via the kexec mechanism.
+
+Booting behaviour can be configured in a number of ways such as boot priority based on device, network-boot behaviour, user permissions, and platform-specific options.
+
+The primary user interface is a ncurses based menu which displays boot options and system information, and provides methods to change configuration options, execute plugins, and drop to a normal shell.
+
+Petitboot is intended to be built as part of a small Linux image, for example one built via tool such as Buildroot. A good example of such an environment is the op-build Buildroot layer used for OpenPOWER: https://github.com/open-power/op-build
diff --git a/doc/platforms.rst b/doc/platforms.rst
new file mode 100644
index 00000000..b4d1d5f0
--- /dev/null
+++ b/doc/platforms.rst
@@ -0,0 +1,16 @@ 
+Platforms
+=========
+
+Petitboot is indented to be platform agnostic and provide at least basic functionality on any platform that supports the kexec mechanism
+
+The 'platform' interface in ``pb-discover`` allows additional support for specific platforms to be built and dynamically enabled based on the running system. The current implementations of this are:
+
+platform-powerpc
+----------------
+
+Intended for "powernv" POWER platforms, which generally means those running the OPAL firmware stack. This implements support for inband IPMI functions (boot overrides, system information, etc), console interface descriptions, and loading and storing parameters from NVRAM flash storage.
+
+platform-arm64
+--------------
+
+A fairly general implementation for basic parameter support for ARM64 platforms, but is generally applicable to platforms that provide an efivarfs interface.