From patchwork Thu Feb 12 03:07:05 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Congyang X-Patchwork-Id: 439004 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 76948140129 for ; Thu, 12 Feb 2015 14:06:22 +1100 (AEDT) Received: from localhost ([::1]:47979 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YLk6e-0003HX-Hd for incoming@patchwork.ozlabs.org; Wed, 11 Feb 2015 22:06:20 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60929) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YLk5k-00024U-9g for qemu-devel@nongnu.org; Wed, 11 Feb 2015 22:05:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YLk5d-00016a-Ct for qemu-devel@nongnu.org; Wed, 11 Feb 2015 22:05:24 -0500 Received: from [59.151.112.132] (port=8994 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YLk5c-00016B-HZ for qemu-devel@nongnu.org; Wed, 11 Feb 2015 22:05:17 -0500 X-IronPort-AV: E=Sophos;i="5.04,848,1406563200"; d="scan'208";a="57476517" Received: from unknown (HELO edo.cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 12 Feb 2015 11:01:25 +0800 Received: from G08CNEXCHPEKD02.g08.fujitsu.local (localhost.localdomain [127.0.0.1]) by edo.cn.fujitsu.com (8.14.3/8.13.1) with ESMTP id t1C34CDH030033; Thu, 12 Feb 2015 11:04:12 +0800 Received: from G08FNSTD140052.g08.fujitsu.local (10.167.226.52) by G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.181.6; Thu, 12 Feb 2015 11:04:58 +0800 From: Wen Congyang To: qemu devel , Kevin Wolf , Stefan Hajnoczi , Paolo Bonzini Date: Thu, 12 Feb 2015 11:07:05 +0800 Message-ID: <1423710438-14377-2-git-send-email-wency@cn.fujitsu.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1423710438-14377-1-git-send-email-wency@cn.fujitsu.com> References: <1423710438-14377-1-git-send-email-wency@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.52] X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Cc: Lai Jiangshan , Jiang Yunhong , Dong Eddie , "Dr. David Alan Gilbert" , Gonglei , Yang Hongyang , zhanghailiang Subject: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Signed-off-by: Wen Congyang Signed-off-by: Paolo Bonzini Signed-off-by: Yang Hongyang Signed-off-by: zhanghailiang Signed-off-by: Gonglei --- docs/block-replication.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/block-replication.txt diff --git a/docs/block-replication.txt b/docs/block-replication.txt new file mode 100644 index 0000000..59150b8 --- /dev/null +++ b/docs/block-replication.txt @@ -0,0 +1,129 @@ +Block replication +---------------------------------------- +Copyright Fujitsu, Corp. 2015 +Copyright (c) 2015 Intel Corporation +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD. + +This work is licensed under the terms of the GNU GPL, version 2 or later. +See the COPYING file in the top-level directory. + +The block replication is used for continuous checkpoints. It is designed +for COLO that Secondary VM is running. It can also be applied for FT/HA +scene that Secondary VM is not running. + +This document gives an overview of block replication's design. + +== Background == +High availability solutions such as micro checkpoint and COLO will do +consecutive checkpoint. The VM state of Primary VM and Secondary VM is +identical right after a VM checkpoint, but becomes different as the VM +executes till the next checkpoint. To support disk contents checkpoint, +the modified disk contents in the Secondary VM must be buffered, and are +only dropped at next checkpoint time. To reduce the network transportation +effort at the time of checkpoint, the disk modification operations of +Primary disk are asynchronously forwarded to the Secondary node. + +== Workflow == +The following is the image of block replication workflow: + + +----------------------+ +------------------------+ + |Primary Write Requests| |Secondary Write Requests| + +----------------------+ +------------------------+ + | | + | (4) + | V + | /-------------\ + | Copy and Forward | | + |---------(1)----------+ | Disk Buffer | + | | | | + | (3) \-------------/ + | speculative ^ + | write through (2) + | | | + V V | + +--------------+ +----------------+ + | Primary Disk | | Secondary Disk | + +--------------+ +----------------+ + + 1) Primary write requests will be copied and forwarded to Secondary + QEMU. + 2) Before Primary write requests are written to Secondary disk, the + original sector content will be read from Secondary disk and + buffered in the Disk buffer, but it will not overwrite the existing + sector content in the Disk buffer. + 3) Primary write requests will be written to Secondary disk. + 4) Secondary write requests will be buffered in the Disk buffer and it + will overwrite the existing sector content in the buffer. + +== Architecture == +We are going to implement COLO block replication from many basic +blocks that are already in QEMU. + + virtio-blk || + ^ || .---------- + | || | Secondary + 1 Quorum || '---------- + / \ || + / \ || + Primary 2 NBD -------> 2 NBD + disk client || server virtio-blk + || ^ ^ +--------. || | | +Primary | || Secondary disk <--------- COLO buffer 3 +--------' || backing + +1) The disk on the primary is represented by a block device with two +children, providing replication between a primary disk and the host that +runs the secondary VM. The read pattern for quorum can be extended to +make the primary always read from the local disk instead of going through +NBD. + +2) The secondary disk receives writes from the primary VM through QEMU's +embedded NBD server (speculative write-through). + +3) The disk on the secondary is represented by a custom block device +("COLO buffer"). The disk buffer's backing image is the secondary disk, +and the disk buffer uses bdrv_add_before_write_notifier to implement +copy-on-write, similar to block/backup.c. + +== New block driver interface == +We add three block driver interfaces to control block replication: +a. bdrv_start_replication() + Start block replication, called in migration/checkpoint thread. + We must call bdrv_start_replication() in secondary QEMU before + calling bdrv_start_replication() in primary QEMU. +b. bdrv_do_checkpoint() + This interface is called after all VM state is transfered to + Secondary QEMU. The Disk buffer will be dropped in this interface. +c. bdrv_stop_replication() + It is called when failover. We will flush the Disk buffer into + Secondary Disk and stop block replication. + +== Usage == +Primary: + -drive if=xxx,driver=quorum,read-pattern=first,\ + children.0.file.filename=1.raw,\ + children.0.driver=raw,\ + children.1.file.driver=nbd+colo,\ + children.1.file.host=xxx,\ + children.1.file.port=xxx,\ + children.1.file.export=xxx,\ + children.1.driver=raw + Note: + 1. NBD Client should not be the first child of quorum. + 2. There should be only one NBD Client. + 3. host is the secondary physical machine's hostname or IP + 4. Each disk must have its own export name. + +Secondary: + -drive if=xxx,driver=blkcolo,export=xxx,\ + backing.file.filename=1.raw,\ + backing.driver=raw + Then run qmp command: + nbd_server_start host:port + Note: + 1. The export name for the same disk must be the same in primary + and secondary QEMU command line + 2. The qmp command nbd_server_start must be run before running the + qmp command migrate on primary QEMU + 3. Don't use nbd_server_start's other options