From patchwork Mon Mar 18 03:19:00 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrhines@linux.vnet.ibm.com X-Patchwork-Id: 228350 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 886872C00BE for ; Mon, 18 Mar 2013 14:20:17 +1100 (EST) Received: from localhost ([::1]:35268 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHQcR-0006EL-Kr for incoming@patchwork.ozlabs.org; Sun, 17 Mar 2013 23:20:15 -0400 Received: from eggs.gnu.org ([208.118.235.92]:44706) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHQbn-0005yt-P8 for qemu-devel@nongnu.org; Sun, 17 Mar 2013 23:19:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UHQbg-0004Xr-Gm for qemu-devel@nongnu.org; Sun, 17 Mar 2013 23:19:35 -0400 Received: from e9.ny.us.ibm.com ([32.97.182.139]:57461) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHQbg-0004XD-Bn for qemu-devel@nongnu.org; Sun, 17 Mar 2013 23:19:28 -0400 Received: from /spool/local by e9.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 17 Mar 2013 23:19:28 -0400 Received: from d01dlp02.pok.ibm.com (9.56.250.167) by e9.ny.us.ibm.com (192.168.1.109) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sun, 17 Mar 2013 23:19:26 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 1C0046E8044 for ; Sun, 17 Mar 2013 23:19:23 -0400 (EDT) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2I3JPok299586 for ; Sun, 17 Mar 2013 23:19:25 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2I3JO0b009306 for ; Sun, 17 Mar 2013 23:19:25 -0400 Received: from mrhinesdev.klabtestbed.com (klinux.watson.ibm.com [9.2.208.21]) by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r2I3JMH5009195; Sun, 17 Mar 2013 23:19:24 -0400 From: mrhines@linux.vnet.ibm.com To: qemu-devel@nongnu.org Date: Sun, 17 Mar 2013 23:19:00 -0400 Message-Id: <1363576743-6146-8-git-send-email-mrhines@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> References: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> X-TM-AS-MML: No X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13031803-7182-0000-0000-000005D4BCE5 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 32.97.182.139 Cc: aliguori@us.ibm.com, mst@redhat.com, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com Subject: [Qemu-devel] [RFC PATCH RDMA support v4: 07/10] connection-establishment for RDMA X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Michael R. Hines" Signed-off-by: Michael R. Hines --- migration-rdma.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 migration-rdma.c diff --git a/migration-rdma.c b/migration-rdma.c new file mode 100644 index 0000000..e1ea055 --- /dev/null +++ b/migration-rdma.c @@ -0,0 +1,205 @@ +/* + * Copyright (C) 2013 Michael R. Hines + * Copyright (C) 2013 Jiuxing Liu + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; under version 2 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ +#include "migration/rdma.h" +#include "qemu-common.h" +#include "migration/migration.h" +#include "migration/qemu-file.h" +#include +#include +#include +#include +#include +#include + +//#define DEBUG_MIGRATION_RDMA + +#ifdef DEBUG_MIGRATION_RDMA +#define DPRINTF(fmt, ...) \ + do { printf("migration-rdma: " fmt, ## __VA_ARGS__); } while (0) +#else +#define DPRINTF(fmt, ...) \ + do { } while (0) +#endif + +static int rdma_accept_incoming_migration(RDMAData *rdma, Error **errp) +{ + int ret; + + ret = qemu_rdma_migrate_listen(rdma, rdma->host, rdma->port); + if (ret) { + qemu_rdma_print("rdma migration: error listening!"); + goto err_rdma_server_wait; + } + + ret = qemu_rdma_alloc_qp(&rdma->rdma_ctx); + if (ret) { + qemu_rdma_print("rdma migration: error allocating qp!"); + goto err_rdma_server_wait; + } + + ret = qemu_rdma_migrate_accept(&rdma->rdma_ctx, NULL, NULL, NULL, 0); + if (ret) { + qemu_rdma_print("rdma migration: error accepting connection!"); + goto err_rdma_server_wait; + } + + ret = qemu_rdma_post_recv_qemu_file(rdma); + if (ret) { + qemu_rdma_print("rdma migration: error posting second qemu file recv!"); + goto err_rdma_server_wait; + } + + ret = qemu_rdma_post_send_remote_info(rdma); + if (ret) { + qemu_rdma_print("rdma migration: error sending remote info!"); + goto err_rdma_server_wait; + } + + ret = qemu_rdma_wait_for_wrid(rdma, RDMA_WRID_SEND_REMOTE_INFO); + if (ret < 0) { + qemu_rdma_print("rdma migration: polling remote info error!"); + goto err_rdma_server_wait; + } + + rdma->total_bytes = 0; + rdma->enabled = 1; + qemu_rdma_dump_gid("server_connect", rdma->rdma_ctx.cm_id); + return 0; + +err_rdma_server_wait: + qemu_rdma_cleanup(rdma); + return -1; + +} + +int rdma_start_incoming_migration(const char * host_port, Error **errp) +{ + RDMAData *rdma = g_malloc0(sizeof(RDMAData)); + QEMUFile *f; + int ret; + + if ((ret = qemu_rdma_data_init(rdma, host_port, errp)) < 0) + return ret; + + ret = qemu_rdma_server_init(rdma, NULL); + + DPRINTF("Starting RDMA-based incoming migration\n"); + + if (!ret) { + DPRINTF("qemu_rdma_server_init success\n"); + ret = qemu_rdma_server_prepare(rdma, NULL); + + if (!ret) { + DPRINTF("qemu_rdma_server_prepare success\n"); + + ret = rdma_accept_incoming_migration(rdma, NULL); + if(!ret) + DPRINTF("qemu_rdma_accept_incoming_migration success\n"); + f = qemu_fopen_rdma(rdma, "rb"); + if (f == NULL) { + fprintf(stderr, "could not qemu_fopen RDMA\n"); + ret = -EIO; + } + + process_incoming_migration(f); + } + } + + return ret; +} + +void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error **errp) +{ + RDMAData *rdma = g_malloc0(sizeof(RDMAData)); + MigrationState *s = opaque; + int ret; + + if (qemu_rdma_data_init(rdma, host_port, errp) < 0) + return; + + ret = qemu_rdma_client_init(rdma, NULL); + if(!ret) { + DPRINTF("qemu_rdma_client_init success\n"); + ret = qemu_rdma_client_connect(rdma, NULL); + + if(!ret) { + s->file = qemu_fopen_rdma(rdma, "wb"); + DPRINTF("qemu_rdma_client_connect success\n"); + migrate_fd_connect(s); + return; + } + } + + migrate_fd_error(s); +} + +size_t save_rdma_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, int cont, size_t size) +{ + int ret; + size_t bytes_sent = 0; + ram_addr_t current_addr; + RDMAData * rdma = migrate_use_rdma(f); + + current_addr = block_offset + offset; + + /* + * Add this page to the current 'chunk'. If the chunk + * is full, an actual RDMA write will occur. + */ + if ((ret = qemu_rdma_write(rdma, current_addr, size)) < 0) { + fprintf(stderr, "rdma migration: write error! %d\n", ret); + return ret; + } + + /* + * Drain the Completion Queue if possible. + * If not, the end of the iteration will do this + * again to make sure we don't overflow the + * request queue. + */ + while (1) { + int ret = qemu_rdma_poll(rdma); + if (ret == RDMA_WRID_NONE) { + break; + } + if (ret < 0) { + fprintf(stderr, "rdma migration: polling error! %d\n", ret); + return ret; + } + } + + bytes_sent += size; + return bytes_sent; +} + +size_t qemu_rdma_fill(void * opaque, uint8_t *buf, int size) +{ + RDMAData * rdma = opaque; + size_t len = 0; + + if(rdma->qemu_file_len) { + DPRINTF("RDMA %" PRId64 " of %d bytes already in buffer\n", + rdma->qemu_file_len, size); + + len = MIN(size, rdma->qemu_file_len); + memcpy(buf, rdma->qemu_file_curr, len); + rdma->qemu_file_curr += len; + rdma->qemu_file_len -= len; + } + + return len; +}