diff mbox series

[ovs-dev,20/22] ovsdb-server: Allow user-provided config files.

Message ID 20231214010431.1664005-21-i.maximets@ovn.org
State Superseded
Delegated to: Ilya Maximets
Headers show
Series [ovs-dev,01/22] ovsdb-server.at: Enbale debug logs in active-backup tests. | expand

Checks

Context Check Description
ovsrobot/apply-robot success apply and check: success
ovsrobot/github-robot-_Build_and_Test success github build: passed
ovsrobot/intel-ovs-compilation success test: success

Commit Message

Ilya Maximets Dec. 14, 2023, 1:04 a.m. UTC 
OVSDB server maintains a temporary file with the current database
configuration for the case it is restarted by a monitor process
after a crash.  On startup the configuration from command line
arguments is stored there in a JSON format, also whenever user
changes the configuration with different UnixCtl commands, those
changes are getting added to the file.  When restarted from the
crash it reads the configuration from the file and continues
with all the necessary remotes and databases.

This change allows it to be an external user-provided file that
OVSDB server will read the configuration from.  The file can be
specified with a --config-file command line argument and it is
mutually exclusive with most other command line arguments that
set up remotes or databases, it is also mutually exclusive with
use of appctl commands that modify same configurations, e.g.
add/remove-db or add/remove-remote.

If the user wants to change the configuration of a running server,
they may change the file and call ovsdb-server/reload appctl.
OVSDB server will open a file, read and parse it, compare the
new configuration with the current one and adjust the running
configuration as needed.  OVSDB server will try to keep existing
databases and connections intact, if the change can be applied
without disrupting the normal operation.

User-provided files are not trustworthy, so extra checks were
added to ensure a correct file format.  If the file cannot be
correctly parsed, e.g. contains invalid JSON, no changes will
be applied and the server will keep using the previous
configuration until the next reload.

If config-file is provided for active-backup databases, permanent
disconnection of one of the backup databases no longer leads to
switching all other databases to 'active'.  Only the disconnected
one will transition, since all of them have their own records in
the configuration file.

With this change, users can run all types of databases within
the same ovsdb-server process at the same time.

Simple configuration may look like this:

 {
    "remotes": {
        "punix:db.sock": {},
        "pssl:6641": {
            "inactivity-probe": 16000,
            "read-only": false,
            "role": "ovn-controller"
        }
    },
    "databases": {
        "conf.db": {},
        "sb.db": {
            "service-model": "clustered"
        },
        "OVN_Northbound": {
            "service-model": "relay",
            "source": {
                "ssl:[fe:::1]:6642,ssl:[fe:::2]:6642": {
                    "max-backoff": 8000,
                    "inactivity-probe": 10000
                }
            }
        }
    }
 }

Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
---
 Documentation/ref/ovsdb.7.rst        |  86 +++++-
 Documentation/topics/ovsdb-relay.rst |  19 ++
 NEWS                                 |   4 +
 ovsdb/ovsdb-server.1.in              |  96 ++++++-
 ovsdb/ovsdb-server.c                 | 384 ++++++++++++++++++++++-----
 5 files changed, 513 insertions(+), 76 deletions(-)

Comments

Frode Nordahl Dec. 14, 2023, 7:57 a.m. UTC | #1 
On Thu, Dec 14, 2023 at 2:05 AM Ilya Maximets <i.maximets@ovn.org> wrote:
>
> OVSDB server maintains a temporary file with the current database
> configuration for the case it is restarted by a monitor process
> after a crash.  On startup the configuration from command line
> arguments is stored there in a JSON format, also whenever user
> changes the configuration with different UnixCtl commands, those
> changes are getting added to the file.  When restarted from the
> crash it reads the configuration from the file and continues
> with all the necessary remotes and databases.
>
> This change allows it to be an external user-provided file that
> OVSDB server will read the configuration from.  The file can be
> specified with a --config-file command line argument and it is
> mutually exclusive with most other command line arguments that
> set up remotes or databases, it is also mutually exclusive with
> use of appctl commands that modify same configurations, e.g.
> add/remove-db or add/remove-remote.
>
> If the user wants to change the configuration of a running server,
> they may change the file and call ovsdb-server/reload appctl.
> OVSDB server will open a file, read and parse it, compare the
> new configuration with the current one and adjust the running
> configuration as needed.  OVSDB server will try to keep existing
> databases and connections intact, if the change can be applied
> without disrupting the normal operation.
>
> User-provided files are not trustworthy, so extra checks were
> added to ensure a correct file format.  If the file cannot be
> correctly parsed, e.g. contains invalid JSON, no changes will
> be applied and the server will keep using the previous
> configuration until the next reload.
>
> If config-file is provided for active-backup databases, permanent
> disconnection of one of the backup databases no longer leads to
> switching all other databases to 'active'.  Only the disconnected
> one will transition, since all of them have their own records in
> the configuration file.
>
> With this change, users can run all types of databases within
> the same ovsdb-server process at the same time.
>
> Simple configuration may look like this:
>
>  {
>     "remotes": {
>         "punix:db.sock": {},
>         "pssl:6641": {
>             "inactivity-probe": 16000,
>             "read-only": false,
>             "role": "ovn-controller"
>         }
>     },
>     "databases": {
>         "conf.db": {},
>         "sb.db": {
>             "service-model": "clustered"

As noted earlier in the series, I think we should omit 'standalone'
and 'clustered' in the vocabulary so it is clear to the user that this
is determined from the on-disk database file and avoid duplication of
effort in keeping the two in sync.
Terry Wilson Jan. 5, 2024, 8:26 p.m. UTC | #2 
On Wed, Dec 13, 2023 at 7:05 PM Ilya Maximets <i.maximets@ovn.org> wrote:

> -/* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
> - * 'config_file', which must have been previously written by save_config(). */
> -static void
> +/* Clears and replaces 'remotes' and 'db_conf' by a configuration read from
> + * 'config_file', which must have been previously written by save_config()
> + * or provided by the user with --config-file.
> + * Returns 'true', if parsing was successful, 'false' otherwise. */
> +static bool
>  load_config(FILE *config_file, struct shash *remotes,
>              struct shash *db_conf, char **sync_from,
>              char **sync_exclude, bool *is_backup)
> @@ -2890,17 +3117,34 @@ load_config(FILE *config_file, struct shash *remotes,
>      struct json *json;

I'm wondering if having an argument for everything we parse out of a
config file might get a little unwieldy down the road. Say we add
configuration of SSL stuff, etc. Maybe it's something we could modify
as it becomes an issue, but it might be nice to have something for
config options that is similar to what we have for registering unixctl
commands or struct ctl_command_syntax. Documentation could be added as
part of the registration/definition of the config option, there could
be a .get() that parses the values out of the json, and a .run() that
gets called after all of the parsing is shown to succeed.

Terry
Ilya Maximets Jan. 8, 2024, 2:16 p.m. UTC | #3 
On 1/5/24 21:26, Terry Wilson wrote:
> On Wed, Dec 13, 2023 at 7:05 PM Ilya Maximets <i.maximets@ovn.org> wrote:
> 
>> -/* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
>> - * 'config_file', which must have been previously written by save_config(). */
>> -static void
>> +/* Clears and replaces 'remotes' and 'db_conf' by a configuration read from
>> + * 'config_file', which must have been previously written by save_config()
>> + * or provided by the user with --config-file.
>> + * Returns 'true', if parsing was successful, 'false' otherwise. */
>> +static bool
>>  load_config(FILE *config_file, struct shash *remotes,
>>              struct shash *db_conf, char **sync_from,
>>              char **sync_exclude, bool *is_backup)
>> @@ -2890,17 +3117,34 @@ load_config(FILE *config_file, struct shash *remotes,
>>      struct json *json;
> 
> I'm wondering if having an argument for everything we parse out of a
> config file might get a little unwieldy down the road. Say we add
> configuration of SSL stuff, etc. Maybe it's something we could modify
> as it becomes an issue, but it might be nice to have something for
> config options that is similar to what we have for registering unixctl
> commands or struct ctl_command_syntax. Documentation could be added as
> part of the registration/definition of the config option, there could
> be a .get() that parses the values out of the json, and a .run() that
> gets called after all of the parsing is shown to succeed.
> 
> Terry
> 

Hi, Terry.  Yes, I agree that the current interface is far from being
ideal, and I actually tried to rework it multiple times while working
on this patch set.  That's one of the reasons it took so long. :)
Unfortunately it ended up with a huge amount of refactoring every time.
The main reason for that, I think, is the fact that the same function
is used for loading the legacy internal configuration file and the new
user-provided file.  And the code around legacy internal configuration
is not easy to adapt.

I think, it would be much easier to do this kind of refactoring once
we deprecate and remove all the appctl commands that can change the
server configuration.  For now I decided to keep the interface as it is
without this patch set to avoid making the patch set even bigger.
I hope that makes sense.

The idea of registering the options with a common parsing logic sounds
interesting though and definitely worth exploring in the future.

Best regards, Ilya Maximets.
Terry Wilson Jan. 10, 2024, 3:06 p.m. UTC | #4 
On Mon, Jan 8, 2024 at 8:17 AM Ilya Maximets <i.maximets@ovn.org> wrote:
>
> On 1/5/24 21:26, Terry Wilson wrote:
> > On Wed, Dec 13, 2023 at 7:05 PM Ilya Maximets <i.maximets@ovn.org> wrote:
> >
> >> -/* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
> >> - * 'config_file', which must have been previously written by save_config(). */
> >> -static void
> >> +/* Clears and replaces 'remotes' and 'db_conf' by a configuration read from
> >> + * 'config_file', which must have been previously written by save_config()
> >> + * or provided by the user with --config-file.
> >> + * Returns 'true', if parsing was successful, 'false' otherwise. */
> >> +static bool
> >>  load_config(FILE *config_file, struct shash *remotes,
> >>              struct shash *db_conf, char **sync_from,
> >>              char **sync_exclude, bool *is_backup)
> >> @@ -2890,17 +3117,34 @@ load_config(FILE *config_file, struct shash *remotes,
> >>      struct json *json;
> >
> > I'm wondering if having an argument for everything we parse out of a
> > config file might get a little unwieldy down the road. Say we add
> > configuration of SSL stuff, etc. Maybe it's something we could modify
> > as it becomes an issue, but it might be nice to have something for
> > config options that is similar to what we have for registering unixctl
> > commands or struct ctl_command_syntax. Documentation could be added as
> > part of the registration/definition of the config option, there could
> > be a .get() that parses the values out of the json, and a .run() that
> > gets called after all of the parsing is shown to succeed.
> >
> > Terry
> >
>
> Hi, Terry.  Yes, I agree that the current interface is far from being
> ideal, and I actually tried to rework it multiple times while working
> on this patch set.  That's one of the reasons it took so long. :)
> Unfortunately it ended up with a huge amount of refactoring every time.
> The main reason for that, I think, is the fact that the same function
> is used for loading the legacy internal configuration file and the new
> user-provided file.  And the code around legacy internal configuration
> is not easy to adapt.
>
> I think, it would be much easier to do this kind of refactoring once
> we deprecate and remove all the appctl commands that can change the
> server configuration.  For now I decided to keep the interface as it is
> without this patch set to avoid making the patch set even bigger.
> I hope that makes sense.
>
> The idea of registering the options with a common parsing logic sounds
> interesting though and definitely worth exploring in the future.
>
> Best regards, Ilya Maximets.

Yeah, I'm all for merging as-is and prettying up as needed.

In case you have entirely too much free time on your hands (ha!),
here's an ini-style config loading/reloading framework I wrote many
many years ago (Warning: I'd only been programming professionally for
~3 years or so at that point). Asterisk was very modular and had over
100 different config files, and many of them used ini files in
completely different ways so it's definitely more generic and overkill
for what we are needing, but there may be some useful ideas [1-5] if
you ignore all of the macro/varargs magic. ;)

The general idea was a per-module `aco_info` structure that defined
the information that was obtained from configs, `aco_type` structures
that defined what parts of the config file matched up to what fields
of the `aco_info` struc, and aco_option_register() which would define
the specific options set under each [category], how they were matched,
what fields in a struct they set, etc. aco_option_register_custom()
just used custom callback functions to handle setting things up.
aco_process_config() would be called on load/reload and would parse
the config file and create an internal 'pending' representation of the
user-defined aco_info and then if parsing succeeded would then
atomically swap the pending and live aco_info pointers. There were
also pre_apply and post_apply callbacks so modules could do special
things if needed during reloads, etc. Prior to the patch modules would
often just modify their internal structures as the config file was
being processed, so if there was a parse error things would just be
left in a state where configs were partially applied.

Anyway, it may or may not be interesting and certainly isn't something
we need to worry about right now. Just food for thought.

Terry

[1] https://github.com/asterisk/asterisk/blob/master/include/asterisk/config_options.h
[2] https://github.com/asterisk/asterisk/blob/master/main/config_options.c
[3] https://github.com/asterisk/asterisk/blob/master/configs/samples/app_skel.conf.sample
[4] https://github.com/asterisk/asterisk/blob/master/apps/app_skel.c
[5] https://github.com/asterisk/asterisk/commit/d54717c39e62f4cc8b290ac4836c4d4469d87c24
diff mbox series

Patch

diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst
index 84b153d24..42e5f4089 100644
--- a/Documentation/ref/ovsdb.7.rst
+++ b/Documentation/ref/ovsdb.7.rst
@@ -155,6 +155,22 @@  standalone database, configure the server to listen on a "connection method"
 that the client can reach, then point the client to that connection method.
 See `Connection Methods`_ below for information about connection methods.
 
+Open vSwitch 3.3 introduced support for configuration files via
+``--config-file`` command line option.  The configuration file for a server
+with a **standalone** database may look like this::
+
+  {
+      "remotes": { "<connection method>": {} },
+      "databases": {
+          "<database file>": {
+              "service-model": "standalone"
+          }
+      }
+  }
+
+The ``"service-model"`` key can be omitted.  In this case ``ovsdb-server``
+will infer the service model from the database file itself.
+
 Active-Backup Database Service Model
 ------------------------------------
 
@@ -177,10 +193,36 @@  database file from the active server.  Then use
 connects to the active server.  At that point, the backup server will fetch a
 copy of the active database and keep it up-to-date until it is killed.
 
+Open vSwitch 3.3 introduced support for configuration files via
+``--config-file`` command line option.  The configuration file for a backup
+server in this case may look like this::
+
+  {
+      "remotes": { "<connection method>": {} },
+      "databases": {
+          "<database file>": {
+              "service-model": "active-backup",
+              "backup": true,
+              "source": {
+                  "<active>": {
+                      "inactivity-probe": <integer>,
+                      "max-backoff": <integer>
+                  }
+              }
+          }
+      }
+  }
+
+All the fields in the ``"<database file>"`` description above are required.
+Options for the ``"<active>"`` connection method (``"inactivity-probe"``, etc.)
+can be omitted.
+
 When the active server in an active-backup server pair fails, an administrator
 can switch the backup server to an active role with the ``ovs-appctl`` command
 ``ovsdb-server/disconnect-active-ovsdb-server``.  Clients then have read/write
-access to the now-active server.  Of course, administrators are slow to respond
+access to the now-active server.  When the ``--config-file`` is in use, the
+same can be achieved by changing the ``"backup"`` value in the file and running
+``ovsdb-server/reload`` command.  Of course, administrators are slow to respond
 compared to software, so in practice external management software detects the
 active server's failure and changes the backup server's role.  For example, the
 "Integration Guide for Centralized Control" in the OVN documentation describes
@@ -236,6 +278,22 @@  To set up a clustered database, first initialize it on a single node by running
 arguments, the ``create-cluster`` command can create an empty database or copy
 a standalone database's contents into the new database.
 
+Open vSwitch 3.3 introduced support for configuration files via
+``--config-file`` command line option.  The configuration file for a server
+with a **clustered** database may look like this::
+
+  {
+      "remotes": { "<connection method>": {} },
+      "databases": {
+          "<database file>": {
+              "service-model": "clustered"
+          }
+      }
+  }
+
+The ``"service-model"`` key can be omitted.  In this case ``ovsdb-server``
+will infer the service model from the database file itself.
+
 To configure a client to use a clustered database, first configure all of the
 servers to listen on a connection method that the client can reach, then point
 the client to all of the servers' connection methods, comma-separated.  See
@@ -505,6 +563,29 @@  server.  ``<relay source>`` could contain a comma-separated list of connection
 methods, e.g. to connect to any server of the clustered database.
 Multiple relay servers could be started for the same relay source.
 
+Open vSwitch 3.3 introduced support for configuration files via
+``--config-file`` command line option.  The configuration file for a relay
+database server in this case may look like this::
+
+  {
+      "remotes": { "<connection method>": {} },
+      "databases": {
+          "<DB_NAME>": {
+              "service-model": "relay",
+              "source": {
+                  "<relay source>": {
+                      "inactivity-probe": <integer>,
+                      "max-backoff": <integer>
+                  }
+              }
+          }
+      }
+  }
+
+Both the ``"service-model"`` and the ``"source"`` are required.  Options for
+the ``"<relay source>"`` connection method (``"inactivity-probe"``, etc.)
+can be omitted.
+
 Since the way relays handle read and write transactions is very similar
 to the clustered model where "cluster" means "set of relay servers connected
 to the same relay source", "follower" means "relay server" and the "leader"
@@ -629,7 +710,8 @@  Creating a Database
 
 Creating and starting up the service for a new database was covered
 separately for each database service model in the `Service
-Models`_ section, above.
+Models`_ section, above.  Single ``ovsdb-server`` process may serve
+any number of databases with different service models at the same time.
 
 Backing Up and Restoring a Database
 -----------------------------------
diff --git a/Documentation/topics/ovsdb-relay.rst b/Documentation/topics/ovsdb-relay.rst
index 50a3c6d07..63ea5329b 100644
--- a/Documentation/topics/ovsdb-relay.rst
+++ b/Documentation/topics/ovsdb-relay.rst
@@ -105,6 +105,25 @@  started like this::
   $ ...
   $ ovsdb-server --remote=ptcp:6642:172.16.0.K relay:OVN_Southbound:$REMOTES
 
+Open vSwitch 3.3 introduced support for configuration files via
+``--config-file`` command line option.  The configuration file for a relay
+database servers in this case may look like this::
+
+  {
+      "remotes": { "ptcp:6642:172.16.0.X": {} },
+      "databases": {
+          "OVN_Southbound": {
+              "service-model": "relay",
+              "source": {
+                  "$REMOTES": {}
+              }
+          }
+      }
+  }
+
+See ``ovsdb-server(1)`` and  ``Relay Service Model`` in ``ovsdb(7)`` for more
+configuration options.
+
 Every relay server could connect to any of the cluster members of their choice,
 fairness of load distribution is achieved by shuffling remotes.
 
diff --git a/NEWS b/NEWS
index 63f2842ae..977e88174 100644
--- a/NEWS
+++ b/NEWS
@@ -6,6 +6,10 @@  Post-v3.2.0
        from older version is supported but it may trigger more leader elections
        during the process, and error logs complaining unrecognized fields may
        be observed on old nodes.
+     * New command line option --config-file that allows a fine control over
+       remotes and database configuration, including setting options for
+       connection methods for relays and active-backup replication.
+       For more details see ovsdb-server(1) and ovsdb(7).
    - ovs-appctl:
      * 'ofproto/trace' now reports OpenFlow rules that make up a conjunctive
        flow match.
diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
index da7a6fd5d..91d6a6e0f 100644
--- a/ovsdb/ovsdb-server.1.in
+++ b/ovsdb/ovsdb-server.1.in
@@ -12,6 +12,7 @@  ovsdb\-server \- Open vSwitch database server
 [\fIdatabase\fR]\&...
 [\fIrelay:schema_name:remote\fR]\&...
 [\fB\-\-remote=\fIremote\fR]\&...
+[\fB\-\-config\-file=\fIfile\fR]
 [\fB\-\-run=\fIcommand\fR]
 .so lib/daemon-syn.man
 .so lib/service-syn.man
@@ -44,6 +45,11 @@  If none of database files or relay databases is specified, the default is
 initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR,
 \fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command.
 .PP
+All types of databases can alternatively be added using a configuration
+file provided via \fB\-\-config\-file\fR option.  This option is mutually
+exclusive with specifying \fIdatabase\fR on the command line.  For a detailed
+description of a configuration file format see \fBovsdb\fR(7).
+.PP
 This OVSDB implementation supports standalone, active-backup, relay and
 clustered database service models, as well as database replication.
 See the Service Models section of \fBovsdb\fR(7) for more information.
@@ -105,6 +111,74 @@  It is an error for \fIcolumn\fR to have another type.
 .IP
 To connect or listen on multiple connection methods, use multiple
 \fB\-\-remote\fR options.
+.IP
+Alternatively, remotes can be specified in a "remotes" section of a
+configuration file, if provided using \fB\-\-config\-file\fR option.
+\fB\-\-config\-file\fR and \fB\-\-remote\fR options are mutually
+exclusive.
+.
+.IP "\fB\-\-config-file=\fIfile\fR"
+Specifies a configuration file for \fBovsdb\-server\fR.  This \fIfile\fR
+can contain connection methods and databases used by the server.
+The \fIfile\fR contains a JSON object with two main elements:
+.RS
+.IP "\fBremotes\fR"
+JSON object that contains a set of connection methods in a following format:
+"\fItarget\fR": { "\fIoption\fR": \fIvalue\fR, ... }.  Where \fItarget\fR
+is in the same format as \fIremote\fR in \fB\-\-remote\fR option.
+\fIoption\fR can be \fBmax-backoff\fR (integer), \fBinactivity-probe\fR
+(integer), \fBread-only\fR (boolean), \fBrole\fR (string) or \fBdscp\fR
+(integer) with their allowed \fIvalue\fRs respectively.  The meaning of these
+\fIoption\fRs is the same as in configuration of \fIremote\fR via a database
+row with \fB\-\-remote\fR option.
+.IP "\fBdatabases\fR"
+JSON object that describes databases that should be added to the
+\fBovsdb\-server\fR in the following format: "\fIname\fR":{ "\fIoption\fR":
+\fIvalue\fR, ... }.  Where \fIname\fR is either a file name of a previously
+created and initialized database or a schema name in case of relay
+databases.  Available \fIoption\fRs are:
+.RS
+.IP "\fBservice-model\fR (string)"
+Describes the service model of this database.  One of: \fBstandalone\fR,
+\fBclustered\fR, \fBactive-backup\fR or \fBrelay\fR.  This option is
+required for all types, except for standalone and clustered.  For these
+databases the service model will be inferred from the file, if not
+specified explicitly.  \fBovsdb-server\fR will refuse to add a database
+if the specified \fBservice-model\fR doesn't match with the provided file.
+.IP "\fBsource\fR (JSON object; active-backup or relay)"
+Describes the connection method to the active database or to the relay
+source.  It is a JSON object with exactly one element in the same format
+as elements of "\fBremotes\fR", except that \fBread-only\fR and \fBrole\fR
+options are not applicable.  E.g. \fB"source": { "unix:db.sock": {
+"inactivity-probe": 10000, "max-backoff": 8000 } }\fR
+.IP "\fBbackup\fR (boolean; active-backup only)"
+If set to \fBtrue\fR, \fBovsdb-server\fR will use this database as a
+backup for the specified \fBsource\fR.  Will be served as an active
+database otherwise.
+.IP "\fBexclude-tables\fR (JSON array of strings; active-backup only)"
+List of table names that should be excluded from replication in backup mode,
+e.g. \fB"exclude-tables": [ "Table_One", "Table_Two" ]\fR.
+.RE
+.RE
+.IP
+Content of the most basic configuration file may look like this:
+\fB{ "remotes": { "pssl:6640": {} }, "databases": { "conf.db": {} } }\fR
+.IP
+Examples of configuration files for different service models can be
+found in in \fBovsdb\fR(7).
+.IP
+\fB\-\-config-file\fR option is mutually exclusive with \fB\-\-remote\fR
+as well as with specifying \fIdatabase\fR on a command line.  It is also
+mutually exclusive with all the \fBActive-Backup Options\fR and all the
+\fBRUNTIME MANAGEMENT COMMANDS\fR that can change the configuration of
+the server in conflict with the content of the file, i.e. all the commands
+that manipulate with remotes and databases.  Read-only commands can still
+be used.
+.IP
+In case of changes in the \fIfile\fR, users should run
+\fBovsdb-server/reload\fR command with \fBovs-appctl\fR(8) in order for
+changes to take effect.
+.RE
 .
 .IP "\fB\-\-run=\fIcommand\fR]"
 Ordinarily \fBovsdb\-server\fR runs forever, or until it is told to
@@ -178,6 +252,8 @@  allow the syncing options to be specified using command line options,
 yet start the server, as the default, active server.  To switch the
 running server to backup mode, use \fBovs-appctl(1)\fR to execute the
 \fBovsdb\-server/connect\-active\-ovsdb\-server\fR command.
+.PP
+These options are mutually exclusive with \fB\-\-config\-file\fR.
 .SS "Public Key Infrastructure Options"
 The options described below for configuring the SSL public key
 infrastructure accept a special syntax for obtaining their
@@ -230,6 +306,8 @@  clients.
 Adds a remote, as if \fB\-\-remote=\fIremote\fR had been specified on
 the \fBovsdb\-server\fR command line.  (If \fIremote\fR is already a
 remote, this command succeeds without changing the configuration.)
+.IP
+Mutually exclusive with \fB\-\-config\-file\fR option.
 .
 .IP "\fBovsdb\-server/remove\-remote \fIremote\fR"
 Removes the specified \fIremote\fR from the configuration, failing
@@ -241,6 +319,8 @@  configuring a \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote.
 (You can remove a database source with \fBovsdb\-server/remove\-remote
 \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR, but not individual
 remotes found indirectly through the database.)
+.IP
+Mutually exclusive with \fB\-\-config\-file\fR option.
 .
 .IP "\fBovsdb\-server/list\-remotes"
 Outputs a list of the currently configured remotes named on
@@ -254,6 +334,8 @@  Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR.  \fIdatabase\fR
 could be a database file or a relay description in the following format:
 \fIrelay:schema_name:remote\fR.  The database file must already have been
 created and initialized using, for example, \fBovsdb\-tool create\fR.
+.IP
+Mutually exclusive with \fB\-\-config\-file\fR option.
 .
 .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR"
 Removes \fIdatabase\fR from the running \fBovsdb\-server\fR.  \fIdatabase\fR
@@ -268,6 +350,8 @@  Any public key infrastructure options specified through this database
 (e.g. \fB\-\-private\-key=db:\fIdatabase,\fR... on the command line)
 will be disabled until another database with the same name is added
 again (with \fBovsdb\-server/add\-db\fR).
+.IP
+Mutually exclusive with \fB\-\-config\-file\fR option.
 .
 .IP "\fBovsdb\-server/list\-dbs"
 Outputs a list of the currently configured databases added either through
@@ -286,6 +370,9 @@  These commands query and update the role of \fBovsdb\-server\fR within
 an active-backup pair of servers.  See \fBActive-Backup Options\fR,
 above, and \fBActive-Backup Database Service Model\fR in
 \fBovsdb\fR(7) for more information.
+.PP
+All \fBActive-Backup Commands\fR that change the state of \fBovsdb\-server\fR
+are mutually exclusive with \fB\-\-config\-file\fR option.
 .
 .IP "\fBovsdb\-server/set\-active\-ovsdb\-server \fIserver"
 Sets  the active \fIserver\fR from which \fBovsdb\-server\fR connects through
@@ -324,11 +411,10 @@  Gets  the  tables  that are currently excluded from synchronization.
 Prints a summary of replication run time information. The \fBstate\fR
 information is always provided, indicating whether the server is running
 in the \fIactive\fR or the \fIbackup\fR mode.
-When running in backup mode, replication connection status, which
-can be either \fIconnecting\fR, \fIreplicating\fR or \fIerror\fR, are shown.
-When the connection is in \fIreplicating\fR state, further output shows
-the list of databases currently replicating, and the tables that are
-excluded.
+For all databases with active-backup service model, replication connection
+status, which can be either \fIconnecting\fR, \fIreplicating\fR or
+\fIerror\fR, are shown.  When the connection is in \fIreplicating\fR state,
+further output shows the tables that are currently excluded from replication.
 .
 .SS "Cluster Commands"
 These commands support the \fBovsdb\-server\fR clustered service model.
diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c
index 9726a8d72..05b91a197 100644
--- a/ovsdb/ovsdb-server.c
+++ b/ovsdb/ovsdb-server.c
@@ -195,6 +195,13 @@  static void add_server_db(struct server_config *);
 static void remove_db(struct server_config *, struct shash_node *db, char *);
 static void close_db(struct server_config *, struct db *, char *);
 
+static struct ovsdb_error *update_schema(struct ovsdb *,
+                                         const struct ovsdb_schema *,
+                                         const struct uuid *txnid,
+                                         bool conversion_with_no_data,
+                                         void *aux)
+    OVS_WARN_UNUSED_RESULT;
+
 static void parse_options(int argc, char *argvp[],
                           struct shash *db_conf, struct shash *remotes,
                           char **unixctl_pathp, char **run_command,
@@ -223,7 +230,7 @@  static void save_config__(FILE *config_file, const struct shash *remotes,
                           const char *sync_from, const char *sync_exclude,
                           bool is_backup);
 static void save_config(struct server_config *);
-static void load_config(FILE *config_file, struct shash *remotes,
+static bool load_config(FILE *config_file, struct shash *remotes,
                         struct shash *db_conf, char **sync_from,
                         char **sync_exclude, bool *is_backup);
 
@@ -263,8 +270,9 @@  ovsdb_server_replication_run(struct server_config *config)
     }
 
     /* If one connection is broken, switch all databases to active,
-     * since they are configured via the same command line / appctl. */
-    if (!all_alive && *config->is_backup) {
+     * if they are configured via the command line / appctl and so have
+     * shared configuration. */
+    if (!config_file_path && !all_alive && *config->is_backup) {
         *config->is_backup = false;
 
         SHASH_FOR_EACH (node, config->all_dbs) {
@@ -513,6 +521,196 @@  free_database_configs(struct shash *db_conf)
     shash_clear(db_conf);
 }
 
+static bool
+service_model_can_convert(enum service_model a, enum service_model b)
+{
+    ovs_assert(a != SM_UNDEFINED);
+
+    if (a == b) {
+        return true;
+    }
+
+    if (b == SM_UNDEFINED) {
+        return a == SM_STANDALONE || a == SM_CLUSTERED;
+    }
+
+    /* Conversion can happen only between standalone and active-backup. */
+    return (a == SM_STANDALONE && b == SM_ACTIVE_BACKUP)
+            || (a == SM_ACTIVE_BACKUP && b == SM_STANDALONE);
+}
+
+static void
+database_update_config(struct server_config *server_config,
+                       struct db *db, const struct db_config *new_conf)
+{
+    struct db_config *conf = db->config;
+    enum service_model model = conf->model;
+
+    /* Stop replicating when transitioning to active or standalone. */
+    if (conf->model == SM_ACTIVE_BACKUP && conf->ab.backup
+        && (new_conf->model == SM_STANDALONE || !new_conf->ab.backup)) {
+        ovsdb_server_replication_remove_db(db);
+    }
+
+    db_config_destroy(conf);
+    conf = db->config = db_config_clone(new_conf);
+
+    if (conf->model == SM_UNDEFINED) {
+        /* We're operating on the same file, the model is the same. */
+        conf->model = model;
+    }
+
+    if (conf->model == SM_RELAY) {
+        ovsdb_relay_add_db(db->db, conf->source, update_schema, server_config,
+                           &conf->options->rpc);
+    }
+    if (conf->model == SM_ACTIVE_BACKUP && conf->ab.backup) {
+        const struct uuid *server_uuid;
+
+        server_uuid = ovsdb_jsonrpc_server_get_uuid(server_config->jsonrpc);
+        replication_set_db(db->db, conf->source, conf->ab.sync_exclude,
+                           server_uuid, &conf->options->rpc);
+    }
+}
+
+static bool
+reconfigure_databases(struct server_config *server_config,
+                      struct shash *db_conf)
+{
+    struct db_config *cur_conf, *new_conf;
+    struct shash_node *node, *conf_node;
+    bool res = true;
+    struct db *db;
+
+    /* Remove databases that are no longer in the configuration or have
+     * incompatible configuration.  Update compatible ones. */
+    SHASH_FOR_EACH_SAFE (node, server_config->all_dbs) {
+        db = node->data;
+
+        if (node->name[0] == '_') {
+            /* Skip internal databases. */
+            continue;
+        }
+
+        cur_conf = db->config;
+        conf_node = shash_find(db_conf, db->filename);
+        new_conf = conf_node ? conf_node->data : NULL;
+
+        if (!new_conf) {
+            remove_db(server_config, node,
+                      xasprintf("database %s removed from configuration",
+                                node->name));
+            continue;
+        }
+        if (!service_model_can_convert(cur_conf->model, new_conf->model)) {
+            remove_db(server_config, node,
+                      xasprintf("service model changed for database %s",
+                                node->name));
+            continue;
+        }
+        database_update_config(server_config, db, new_conf);
+
+        db_config_destroy(new_conf);
+        shash_delete(db_conf, conf_node);
+    }
+
+    /* Create new databases. */
+    SHASH_FOR_EACH (node, db_conf) {
+        struct ovsdb_error *error = open_db(server_config,
+                                            node->name, node->data);
+        if (error) {
+            char *s = ovsdb_error_to_string_free(error);
+
+            VLOG_WARN("failed to open database '%s': %s", node->name, s);
+            free(s);
+            res = false;
+        }
+        db_config_destroy(node->data);
+    }
+    shash_clear(db_conf);
+
+    return res;
+}
+
+static bool
+reconfigure_ovsdb_server(struct server_config *server_config)
+{
+    char *sync_from = NULL, *sync_exclude = NULL;
+    bool is_backup = false;
+    struct shash remotes;
+    struct shash db_conf;
+    bool res = true;
+
+    FILE *file = NULL;
+
+    if (config_file_path) {
+        file = fopen(config_file_path, "r+b");
+        if (!file) {
+            VLOG_ERR("failed to open configuration file '%s': %s",
+                     config_file_path, ovs_strerror(errno));
+            return false;
+        } else {
+            VLOG_INFO("loading configuration from '%s'", config_file_path);
+        }
+    } else {
+        file = server_config->config_tmpfile;
+    }
+    ovs_assert(file);
+
+    shash_init(&remotes);
+    shash_init(&db_conf);
+
+    if (!load_config(file, &remotes, &db_conf,
+                     &sync_from, &sync_exclude, &is_backup)) {
+        if (config_file_path) {
+            VLOG_WARN("failed to load configuration from %s",
+                      config_file_path);
+        } else {
+            VLOG_FATAL("failed to load configuration from a temporary file");
+        }
+        res = false;
+        goto exit_close;
+    }
+
+    /* Parsing was successful.  Update the server configuration. */
+    shash_swap(server_config->remotes, &remotes);
+    free(*server_config->sync_from);
+    *server_config->sync_from = sync_from;
+    free(*server_config->sync_exclude);
+    *server_config->sync_exclude = sync_exclude;
+    *server_config->is_backup = is_backup;
+
+    if (!reconfigure_databases(server_config, &db_conf)) {
+        VLOG_WARN("failed to configure databases");
+        res = false;
+    }
+
+    char *error = reconfigure_remotes(server_config->jsonrpc,
+                                      server_config->all_dbs,
+                                      server_config->remotes);
+    if (error) {
+        VLOG_WARN("failed to configure remotes: %s", error);
+        res = false;
+    } else {
+        error = reconfigure_ssl(server_config->all_dbs);
+        if (error) {
+            VLOG_WARN("failed to configure SSL: %s", error);
+            res = false;
+        }
+    }
+    free(error);
+
+exit_close:
+    if (config_file_path) {
+        fclose(file);
+    }
+    free_remotes(&remotes);
+    free_database_configs(&db_conf);
+    shash_destroy(&remotes);
+    shash_destroy(&db_conf);
+    return res;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -527,13 +725,22 @@  main(int argc, char *argv[])
     struct process *run_process;
     bool exiting;
     int retval;
-    FILE *config_tmpfile;
-    struct server_config server_config;
+    FILE *config_tmpfile = NULL;
     struct shash all_dbs;
     struct shash_node *node;
     int replication_probe_interval = REPLICATION_DEFAULT_PROBE_INTERVAL;
     int relay_source_probe_interval = RELAY_SOURCE_DEFAULT_PROBE_INTERVAL;
 
+    struct server_config server_config = {
+        .remotes = &remotes,
+        .all_dbs = &all_dbs,
+        .sync_from = &sync_from,
+        .sync_exclude = &sync_exclude,
+        .is_backup = &is_backup,
+        .replication_probe_interval = &replication_probe_interval,
+        .relay_source_probe_interval = &relay_source_probe_interval,
+    };
+
     ovs_cmdl_proctitle_init(argc, argv);
     set_program_name(argv[0]);
     service_start(&argc, &argv);
@@ -548,64 +755,39 @@  main(int argc, char *argv[])
 
     daemon_become_new_user(false, false);
 
-    /* Create and initialize 'config_tmpfile' as a temporary file to hold
-     * ovsdb-server's most basic configuration, and then save our initial
-     * configuration to it.  When --monitor is used, this preserves the effects
-     * of ovs-appctl commands such as ovsdb-server/add-remote (which saves the
-     * new configuration) across crashes. */
-    config_tmpfile = tmpfile();
-    if (!config_tmpfile) {
-        ovs_fatal(errno, "failed to create temporary file");
+    if (!config_file_path) {
+         /* Create and initialize 'config_tmpfile' as a temporary file to hold
+         * ovsdb-server's most basic configuration, and then save our initial
+         * configuration to it.  When --monitor is used, this preserves the
+         * effects of ovs-appctl commands such as ovsdb-server/add-remote
+         * (which saves the new configuration) across crashes. */
+        config_tmpfile = tmpfile();
+        if (!config_tmpfile) {
+            ovs_fatal(errno, "failed to create temporary file");
+        }
+        server_config.config_tmpfile = config_tmpfile;
+        save_config__(config_tmpfile, &remotes, &db_conf, sync_from,
+                      sync_exclude, is_backup);
     }
 
-    server_config.remotes = &remotes;
-    server_config.config_tmpfile = config_tmpfile;
-
-    save_config__(config_tmpfile, &remotes, &db_conf, sync_from,
-                  sync_exclude, is_backup);
     free_remotes(&remotes);
     free_database_configs(&db_conf);
 
     daemonize_start(false, false);
 
-    /* Load the saved config. */
-    load_config(config_tmpfile, &remotes, &db_conf, &sync_from,
-                &sync_exclude, &is_backup);
-
-    /* Start ovsdb jsonrpc server. When running as a backup server,
-     * jsonrpc connections are read only. Otherwise, both read
-     * and write transactions are allowed.  */
-    jsonrpc = ovsdb_jsonrpc_server_create(is_backup);
+    perf_counters_init();
 
-    shash_init(&all_dbs);
-    server_config.all_dbs = &all_dbs;
+    /* Start ovsdb jsonrpc server.  Both read and write transactions are
+     * allowed by default, individual remotes and databases will be configured
+     * as read-only, if necessary. */
+    jsonrpc = ovsdb_jsonrpc_server_create(false);
     server_config.jsonrpc = jsonrpc;
-    server_config.sync_from = &sync_from;
-    server_config.sync_exclude = &sync_exclude;
-    server_config.is_backup = &is_backup;
-    server_config.replication_probe_interval = &replication_probe_interval;
-    server_config.relay_source_probe_interval = &relay_source_probe_interval;
-
-    perf_counters_init();
 
-    SHASH_FOR_EACH (node, &db_conf) {
-        struct ovsdb_error *error = open_db(&server_config,
-                                            node->name, node->data);
-        if (error) {
-            char *s = ovsdb_error_to_string_free(error);
-            ovs_fatal(0, "%s", s);
-        }
-        db_config_destroy(node->data);
-    }
-    shash_clear(&db_conf);
+    shash_init(&all_dbs);
     add_server_db(&server_config);
 
-    char *error = reconfigure_remotes(jsonrpc, &all_dbs, &remotes);
-    if (!error) {
-        error = reconfigure_ssl(&all_dbs);
-    }
-    if (error) {
-        ovs_fatal(0, "%s", error);
+    if (!reconfigure_ovsdb_server(&server_config)) {
+        ovs_fatal(0, "server configuration failed");
     }
 
     retval = unixctl_server_create(unixctl_path, &unixctl);
@@ -2057,14 +2239,21 @@  ovsdb_server_reconnect(struct unixctl_conn *conn, int argc OVS_UNUSED,
  * 'config_file_path', read it and sync the runtime configuration with it. */
 static void
 ovsdb_server_reload(struct unixctl_conn *conn, int argc OVS_UNUSED,
-                    const char *argv[] OVS_UNUSED, void *config_ OVS_UNUSED)
+                    const char *argv[] OVS_UNUSED, void *config_)
 {
+    struct server_config *config = config_;
+
     if (!config_file_path) {
         unixctl_command_reply_error(conn,
             "Configuration file was not specified on command line");
-    } else {
+        return;
+    }
+
+    if (!reconfigure_ovsdb_server(config)) {
         unixctl_command_reply_error(conn,
-            "Configuration file support is not implemented yet");
+            "Configuration failed.  See the log file for details.");
+    } else {
+        unixctl_command_reply(conn, NULL);
     }
 }
 
@@ -2739,6 +2928,10 @@  save_config(struct server_config *config)
     struct shash_node *node;
     struct shash db_conf;
 
+    if (config_file_path) {
+        return;
+    }
+
     shash_init(&db_conf);
     SHASH_FOR_EACH (node, config->all_dbs) {
         struct db *db = node->data;
@@ -2755,7 +2948,7 @@  save_config(struct server_config *config)
     shash_destroy(&db_conf);
 }
 
-static void
+static bool
 remotes_from_json(struct shash *remotes, const struct json *json)
 {
     struct ovsdb_jsonrpc_options *options;
@@ -2765,14 +2958,31 @@  remotes_from_json(struct shash *remotes, const struct json *json)
     free_remotes(remotes);
 
     ovs_assert(json);
-    ovs_assert(json->type == JSON_OBJECT);
+    if (json->type == JSON_NULL) {
+        return true;
+    }
+    if (json->type != JSON_OBJECT) {
+        VLOG_WARN("config: 'remotes' is not a JSON object");
+        return false;
+    }
 
     object = json_object(json);
     SHASH_FOR_EACH (node, object) {
         options = ovsdb_jsonrpc_default_options(node->name);
-        ovsdb_jsonrpc_options_update_from_json(options, node->data, false);
         shash_add(remotes, node->name, options);
+
+        json = node->data;
+        if (json->type == JSON_OBJECT) {
+            ovsdb_jsonrpc_options_update_from_json(options, node->data, false);
+        } else if (json->type != JSON_NULL) {
+            VLOG_WARN("%s: JSON-RPC options are not a JSON object or null",
+                      node->name);
+            free_remotes(remotes);
+            return false;
+        }
     }
+
+    return true;
 }
 
 static struct db_config *
@@ -2783,6 +2993,12 @@  db_config_from_json(const char *name, const struct json *json)
     struct ovsdb_parser parser;
     struct ovsdb_error *error;
 
+    ovs_assert(json);
+    if (json->type == JSON_NULL) {
+        conf->model = SM_UNDEFINED;
+        return conf;
+    }
+
     ovsdb_parser_init(&parser, json, "database %s", name);
 
     model = ovsdb_parser_member(&parser, "service-model",
@@ -2859,7 +3075,7 @@  db_config_from_json(const char *name, const struct json *json)
 }
 
 
-static void
+static bool
 databases_from_json(struct shash *db_conf, const struct json *json)
 {
     const struct shash_node *node;
@@ -2868,7 +3084,12 @@  databases_from_json(struct shash *db_conf, const struct json *json)
     free_database_configs(db_conf);
 
     ovs_assert(json);
-    ovs_assert(json->type == JSON_OBJECT);
+    if (json->type == JSON_NULL) {
+        return true;
+    }
+    if (json->type != JSON_OBJECT) {
+        VLOG_WARN("config: 'databases' is not a JSON object or null");
+    }
 
     object = json_object(json);
     SHASH_FOR_EACH (node, object) {
@@ -2876,13 +3097,19 @@  databases_from_json(struct shash *db_conf, const struct json *json)
 
         if (conf) {
             shash_add(db_conf, node->name, conf);
+        } else {
+            free_database_configs(db_conf);
+            return false;
         }
     }
+    return true;
 }
 
-/* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
- * 'config_file', which must have been previously written by save_config(). */
-static void
+/* Clears and replaces 'remotes' and 'db_conf' by a configuration read from
+ * 'config_file', which must have been previously written by save_config()
+ * or provided by the user with --config-file.
+ * Returns 'true', if parsing was successful, 'false' otherwise. */
+static bool
 load_config(FILE *config_file, struct shash *remotes,
             struct shash *db_conf, char **sync_from,
             char **sync_exclude, bool *is_backup)
@@ -2890,17 +3117,34 @@  load_config(FILE *config_file, struct shash *remotes,
     struct json *json;
 
     if (fseek(config_file, 0, SEEK_SET) != 0) {
-        VLOG_FATAL("seek failed in temporary file (%s)", ovs_strerror(errno));
+        VLOG_WARN("config: file seek failed (%s)", ovs_strerror(errno));
+        return false;
     }
     json = json_from_stream(config_file);
     if (json->type == JSON_STRING) {
-        VLOG_FATAL("reading json failed (%s)", json_string(json));
+        VLOG_WARN("config: reading JSON failed (%s)", json_string(json));
+        json_destroy(json);
+        return false;
+    }
+    if (json->type != JSON_OBJECT) {
+        VLOG_WARN("configuration in a file must be a JSON object");
+        json_destroy(json);
+        return false;
     }
-    ovs_assert(json->type == JSON_OBJECT);
 
-    remotes_from_json(remotes, shash_find_data(json_object(json), "remotes"));
-    databases_from_json(db_conf,
-                        shash_find_data(json_object(json), "databases"));
+    if (!remotes_from_json(remotes,
+                           shash_find_data(json_object(json), "remotes"))) {
+        VLOG_WARN("config: failed to parse 'remotes'");
+        json_destroy(json);
+        return false;
+    }
+    if (!databases_from_json(db_conf, shash_find_data(json_object(json),
+                                                      "databases"))) {
+        VLOG_WARN("config: failed to parse 'databases'");
+        free_remotes(remotes);
+        json_destroy(json);
+        return false;
+    }
 
     struct json *string;
     string = shash_find_data(json_object(json), "sync_from");
@@ -2911,7 +3155,9 @@  load_config(FILE *config_file, struct shash *remotes,
     free(*sync_exclude);
     *sync_exclude = string ? xstrdup(json_string(string)) : NULL;
 
-    *is_backup = json_boolean(shash_find_data(json_object(json), "is_backup"));
+    struct json *boolean = shash_find_data(json_object(json), "is_backup");
+    *is_backup = boolean ? json_boolean(boolean) : false;
 
     json_destroy(json);
+    return true;
 }