From patchwork Wed Jul 23 14:25:32 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Hongyang X-Patchwork-Id: 372969 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id B116E140087 for ; Thu, 24 Jul 2014 00:34:32 +1000 (EST) Received: from localhost ([::1]:45432 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X9xck-0004Wg-Mu for incoming@patchwork.ozlabs.org; Wed, 23 Jul 2014 10:34:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48274) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X9xUf-00076v-4S for qemu-devel@nongnu.org; Wed, 23 Jul 2014 10:26:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1X9xUX-0005vt-IS for qemu-devel@nongnu.org; Wed, 23 Jul 2014 10:26:08 -0400 Received: from [59.151.112.132] (port=39675 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X9xUW-0005sB-GX for qemu-devel@nongnu.org; Wed, 23 Jul 2014 10:26:01 -0400 X-IronPort-AV: E=Sophos;i="5.00,931,1396972800"; d="scan'208";a="33686785" Received: from unknown (HELO edo.cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 23 Jul 2014 22:23:12 +0800 Received: from G08CNEXCHPEKD03.g08.fujitsu.local (localhost.localdomain [127.0.0.1]) by edo.cn.fujitsu.com (8.14.3/8.13.1) with ESMTP id s6NEPvPq021563; Wed, 23 Jul 2014 22:25:57 +0800 Received: from localhost (10.167.226.223) by G08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.181.6; Wed, 23 Jul 2014 22:26:03 +0800 From: Yang Hongyang To: Date: Wed, 23 Jul 2014 22:25:32 +0800 Message-ID: <1406125538-27992-12-git-send-email-yanghy@cn.fujitsu.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1406125538-27992-1-git-send-email-yanghy@cn.fujitsu.com> References: <1406125538-27992-1-git-send-email-yanghy@cn.fujitsu.com> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Cc: kvm@vger.kernel.org, GuiJianfeng@cn.fujitsu.com, eddie.dong@intel.com, dgilbert@redhat.com, mrhines@linux.vnet.ibm.com, Yang Hongyang Subject: [Qemu-devel] [RFC PATCH 11/17] COLO ctl: implement colo checkpoint protocol X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org implement colo checkpoint protocol. Checkpoint synchronzing points. Primary Secondary NEW @ Suspend SUSPENDED @ Suspend&Save state SEND @ Send state Receive state RECEIVED @ Flush network Load state LOADED @ Resume Resume Start Comparing NOTE: 1) '@' who sends the message 2) Every sync-point is synchronized by two sides with only one handshake(single direction) for low-latency. If more strict synchronization is required, a opposite direction sync-point should be added. 3) Since sync-points are single direction, the remote side may go forward a lot when this side just receives the sync-point. Signed-off-by: Yang Hongyang --- migration-colo.c | 268 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 262 insertions(+), 6 deletions(-) diff --git a/migration-colo.c b/migration-colo.c index 2699e77..a708872 100644 --- a/migration-colo.c +++ b/migration-colo.c @@ -24,6 +24,41 @@ */ #define CHKPOINT_TIMER 10000 +enum { + COLO_READY = 0x46, + + /* + * Checkpoint synchronzing points. + * + * Primary Secondary + * NEW @ + * Suspend + * SUSPENDED @ + * Suspend&Save state + * SEND @ + * Send state Receive state + * RECEIVED @ + * Flush network Load state + * LOADED @ + * Resume Resume + * + * Start Comparing + * NOTE: + * 1) '@' who sends the message + * 2) Every sync-point is synchronized by two sides with only + * one handshake(single direction) for low-latency. + * If more strict synchronization is required, a opposite direction + * sync-point should be added. + * 3) Since sync-points are single direction, the remote side may + * go forward a lot when this side just receives the sync-point. + */ + COLO_CHECKPOINT_NEW, + COLO_CHECKPOINT_SUSPENDED, + COLO_CHECKPOINT_SEND, + COLO_CHECKPOINT_RECEIVED, + COLO_CHECKPOINT_LOADED, +}; + static QEMUBH *colo_bh; bool colo_supported(void) @@ -185,30 +220,161 @@ static const QEMUFileOps colo_read_ops = { .close = colo_close, }; +/* colo checkpoint control helper */ +static bool is_master(void); +static bool is_slave(void); + +static void ctl_error_handler(void *opaque, int err) +{ + if (is_slave()) { + /* TODO: determine whether we need to failover */ + /* FIXME: we will not failover currently, just kill slave */ + error_report("error: colo transmission failed!\n"); + exit(1); + } else if (is_master()) { + /* Master still alive, do not failover */ + error_report("error: colo transmission failed!\n"); + return; + } else { + error_report("COLO: Unexpected error happend!\n"); + exit(EXIT_FAILURE); + } +} + +static int colo_ctl_put(QEMUFile *f, uint64_t request) +{ + int ret = 0; + + qemu_put_be64(f, request); + qemu_fflush(f); + + ret = qemu_file_get_error(f); + if (ret < 0) { + ctl_error_handler(f, ret); + return 1; + } + + return ret; +} + +static int colo_ctl_get_value(QEMUFile *f, uint64_t *value) +{ + int ret = 0; + uint64_t temp; + + temp = qemu_get_be64(f); + + ret = qemu_file_get_error(f); + if (ret < 0) { + ctl_error_handler(f, ret); + return 1; + } + + *value = temp; + return 0; +} + +static int colo_ctl_get(QEMUFile *f, uint64_t require) +{ + int ret; + uint64_t value; + + ret = colo_ctl_get_value(f, &value); + if (ret) { + return ret; + } + + if (value != require) { + error_report("unexpected state received!\n"); + exit(1); + } + + return ret; +} + /* save */ -static __attribute__((unused)) bool is_master(void) +static bool is_master(void) { MigrationState *s = migrate_get_current(); return (s->state == MIG_STATE_COLO); } +static int do_colo_transaction(MigrationState *s, QEMUFile *control, + QEMUFile *trans) +{ + int ret; + + ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW); + if (ret) { + goto out; + } + + ret = colo_ctl_get(control, COLO_CHECKPOINT_SUSPENDED); + if (ret) { + goto out; + } + + /* TODO: suspend and save vm state to colo buffer */ + + ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND); + if (ret) { + goto out; + } + + /* TODO: send vmstate to slave */ + + ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED); + if (ret) { + goto out; + } + + /* TODO: Flush network etc. */ + + ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED); + if (ret) { + goto out; + } + + /* TODO: resume master */ + +out: + return ret; +} + static void *colo_thread(void *opaque) { MigrationState *s = opaque; int dev_hotplug = qdev_hotplug, wait_cp = 0; int64_t start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST); int64_t current_time; + QEMUFile *colo_control = NULL, *colo_trans = NULL; + int ret; if (colo_compare_init() < 0) { error_report("Init colo compare error\n"); goto out; } + colo_control = qemu_fopen_socket(qemu_get_fd(s->file), "rb"); + if (!colo_control) { + error_report("open colo_control failed\n"); + goto out; + } + qdev_hotplug = 0; colo_buffer_init(); + /* + * Wait for slave finish loading vm states and enter COLO + * restore. + */ + ret = colo_ctl_get(colo_control, COLO_READY); + if (ret) { + goto out; + } + while (s->state == MIG_STATE_COLO) { /* wait for a colo checkpoint */ wait_cp = colo_compare(); @@ -230,13 +396,33 @@ static void *colo_thread(void *opaque) /* start a colo checkpoint */ - /*TODO: COLO save */ + /* open colo buffer for write */ + colo_trans = qemu_fopen_ops(&colo_buffer, &colo_write_ops); + if (!colo_trans) { + error_report("open colo buffer failed\n"); + goto out; + } + if (do_colo_transaction(s, colo_control, colo_trans)) { + goto out; + } + + qemu_fclose(colo_trans); + colo_trans = NULL; start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST); } out: + if (colo_trans) { + qemu_fclose(colo_trans); + } + colo_buffer_destroy(); + + if (colo_control) { + qemu_fclose(colo_control); + } + colo_compare_destroy(); if (s->state != MIG_STATE_ERROR) { @@ -281,7 +467,7 @@ void colo_init_checkpointer(MigrationState *s) static Coroutine *colo; -static __attribute__((unused)) bool is_slave(void) +static bool is_slave(void) { return colo != NULL; } @@ -293,13 +479,32 @@ static __attribute__((unused)) bool is_slave(void) */ static int slave_wait_new_checkpoint(QEMUFile *f) { - /* TODO: wait checkpoint start command from master */ - return 1; + int fd = qemu_get_fd(f); + int ret; + uint64_t cmd; + + yield_until_fd_readable(fd); + + ret = colo_ctl_get_value(f, &cmd); + if (ret) { + return 1; + } + + if (cmd == COLO_CHECKPOINT_NEW) { + return 0; + } else { + /* Unexpected data received */ + ctl_error_handler(f, ret); + return 1; + } } void colo_process_incoming_checkpoints(QEMUFile *f) { + int fd = qemu_get_fd(f); int dev_hotplug = qdev_hotplug; + QEMUFile *ctl = NULL; + int ret; if (!restore_use_colo()) { return; @@ -310,18 +515,69 @@ void colo_process_incoming_checkpoints(QEMUFile *f) colo = qemu_coroutine_self(); assert(colo != NULL); + ctl = qemu_fopen_socket(fd, "wb"); + if (!ctl) { + error_report("can't open incoming channel\n"); + goto out; + } + colo_buffer_init(); + ret = colo_ctl_put(ctl, COLO_READY); + if (ret) { + goto out; + } + + /* TODO: in COLO mode, slave is runing, so start the vm */ + while (true) { if (slave_wait_new_checkpoint(f)) { break; } - /* TODO: COLO restore */ + /* start colo checkpoint */ + + /* TODO: suspend guest */ + + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_SUSPENDED); + if (ret) { + goto out; + } + + /* TODO: open colo buffer for read */ + + ret = colo_ctl_get(f, COLO_CHECKPOINT_SEND); + if (ret) { + goto out; + } + + /* TODO: read migration data into colo buffer */ + + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_RECEIVED); + if (ret) { + goto out; + } + + /* TODO: load vm state */ + + ret = colo_ctl_put(ctl, COLO_CHECKPOINT_LOADED); + if (ret) { + goto out; + } + + /* TODO: resume guest */ + + /* TODO: close colo buffer */ } +out: colo_buffer_destroy(); colo = NULL; + + if (ctl) { + qemu_fclose(ctl); + } + restore_exit_colo(); qdev_hotplug = dev_hotplug;