From patchwork Wed Aug 5 15:33:22 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Graf X-Patchwork-Id: 30800 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by bilbo.ozlabs.org (Postfix) with ESMTPS id D3B3BB7D17 for ; Thu, 6 Aug 2009 01:34:00 +1000 (EST) Received: from localhost ([127.0.0.1]:35057 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MYiV1-0005J3-Vq for incoming@patchwork.ozlabs.org; Wed, 05 Aug 2009 11:33:56 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MYiUb-0005Ix-O2 for qemu-devel@nongnu.org; Wed, 05 Aug 2009 11:33:29 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MYiUX-0005Ct-ON for qemu-devel@nongnu.org; Wed, 05 Aug 2009 11:33:29 -0400 Received: from [199.232.76.173] (port=53975 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MYiUX-0005CS-Jf for qemu-devel@nongnu.org; Wed, 05 Aug 2009 11:33:25 -0400 Received: from cantor.suse.de ([195.135.220.2]:42893 helo=mx1.suse.de) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1MYiUW-00050n-Re for qemu-devel@nongnu.org; Wed, 05 Aug 2009 11:33:25 -0400 Received: from relay2.suse.de (relay-ext.suse.de [195.135.221.8]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.suse.de (Postfix) with ESMTP id ECA2193717 for ; Wed, 5 Aug 2009 17:33:22 +0200 (CEST) From: Alexander Graf To: qemu-devel@nongnu.org Date: Wed, 5 Aug 2009 17:33:22 +0200 Message-Id: <1249486402-10824-1-git-send-email-agraf@suse.de> X-Mailer: git-send-email 1.6.0.2 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.4-2.6 Subject: [Qemu-devel] [PATCH] Add tar container format X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Tar is a very widely used format to store data in. Sometimes people even put virtual machine images in there. So it makes sense for qemu to be able to read from tar files. I implemented a written from scratch reader that also knows about the GNU sparse format, which is what pigz creates. This version checks for filenames that end on well-known extensions. The logic could be changed to search for filenames given on the command line, but that would require changes to more parts of qemu. The tar reader in conjunctiuon with dzip gives us the chance to download tar'ed up virtual machine images (even via http) and instantly make use of them. For that we still need to enable the qemu blockery to support stacking though. Signed-off-by: Alexander Graf --- Makefile | 2 +- block/tar.c | 326 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletions(-) create mode 100644 block/tar.c diff --git a/Makefile b/Makefile index 288190d..3183e71 100644 --- a/Makefile +++ b/Makefile @@ -73,7 +73,7 @@ block-obj-$(CONFIG_AIO) += posix-aio-compat.o block-nested-y += cow.o qcow.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat.o block-nested-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o -block-nested-y += parallels.o nbd.o dictzip.o +block-nested-y += parallels.o nbd.o dictzip.o tar.o block-nested-$(CONFIG_WIN32) += raw-win32.o block-nested-$(CONFIG_POSIX) += raw-posix.o block-nested-$(CONFIG_CURL) += curl.o diff --git a/block/tar.c b/block/tar.c new file mode 100644 index 0000000..2c965cc --- /dev/null +++ b/block/tar.c @@ -0,0 +1,326 @@ +/* + * Tar block driver + * + * Copyright (c) 2009 Alexander Graf + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +#include "qemu-common.h" +#include "block_int.h" + +// #define DEBUG + +#ifdef DEBUG +#define dprintf(fmt, ...) do { printf("tar: " fmt, ## __VA_ARGS__); } while (0) +#else +#define dprintf(fmt, ...) do { } while (0) +#endif + +#define SECTOR_SIZE 512 + +#define POSIX_TAR_MAGIC "ustar" +#define OFFS_LENGTH 0x7c +#define OFFS_TYPE 0x9c +#define OFFS_MAGIC 0x101 + +#define OFFS_S_SP 0x182 +#define OFFS_S_EXT 0x1e2 +#define OFFS_S_LENGTH 0x1e3 +#define OFFS_SX_EXT 0x1f8 + +typedef struct SparseCache { + uint64_t start; + uint64_t end; +} SparseCache; + +typedef struct BDRVTarState { + BlockDriverState *hd; + size_t file_sec; + uint64_t file_len; + SparseCache *sparse; + int sparse_num; + uint64_t last_end; +} BDRVTarState; + +static int tar_probe(const uint8_t *buf, int buf_size, const char *filename) +{ + if (buf_size < OFFS_MAGIC + 5) + return 0; + + /* we only support newer tar */ + if (!strncmp((char*)buf + OFFS_MAGIC, POSIX_TAR_MAGIC, 5)) + return 100; + + return 0; +} + +static int str_ends(char *str, const char *end) +{ + int end_len = strlen(end); + int str_len = strlen(str); + + if (str_len < end_len) + return 0; + + return !strncmp(str + str_len - end_len, end, end_len); +} + +static int is_target_file(BlockDriverState *bs, char *filename) +{ + int retval = 0; + + if (str_ends(filename, ".raw")) + retval = 1; + + if (str_ends(filename, ".qcow")) + retval = 1; + + if (str_ends(filename, ".qcow2")) + retval = 1; + + if (str_ends(filename, ".vmdk")) + retval = 1; + + dprintf("does filename %s match? %s\n", filename, retval ? "yes" : "no"); + return retval; +} + +static uint64_t tar2u64(char *ptr) +{ + uint64_t retval; + char oldend = ptr[12]; + + ptr[12] = '\0'; + if (*ptr & 0x80) + retval = be64_to_cpu(*(uint64_t *)ptr); + else + retval = strtol(ptr, NULL, 8); + + ptr[12] = oldend; + + dprintf("Convert %s -> %#lx\n", ptr, retval); + return retval; +} + +static void tar_sparse(BDRVTarState *s, uint64_t offs, uint64_t len) +{ + SparseCache *sparse; + + if (!len) + return; + if (!(offs - s->last_end)) { + s->last_end += len; + return; + } + if (s->last_end > offs) + return; + + dprintf("Last chunk until %lx new chunk at %lx\n", s->last_end, offs); + + s->sparse = qemu_realloc(s->sparse, (s->sparse_num + 1) * sizeof(SparseCache)); + sparse = &s->sparse[s->sparse_num]; + sparse->start = s->last_end; + sparse->end = offs; + s->last_end = offs + len; + s->sparse_num++; + dprintf("Sparse at %lx end=%lx\n", sparse->start, + sparse->end); +} + +static int tar_open(BlockDriverState *bs, const char *filename, int flags) +{ + BDRVTarState *s = bs->opaque; + char header[SECTOR_SIZE]; + char *magic; + size_t header_offs = 0; + int ret; + + ret = bdrv_file_open(&s->hd, filename, flags); + if (ret < 0) + return ret; + + /* Search the file for an image */ + + do { + /* tar header */ + if (bdrv_pread(s->hd, header_offs, header, SECTOR_SIZE) != SECTOR_SIZE) + goto fail; + + if ((header_offs > 1) && !header[0]) { + fprintf(stderr, "Tar: No image file found in archive\n"); + goto fail; + } + + magic = &header[OFFS_MAGIC]; + if (strncmp(magic, POSIX_TAR_MAGIC, 5)) { + fprintf(stderr, "Tar: Invalid magic: %s\n", magic); + goto fail; + } + + dprintf("file type: %c\n", header[OFFS_TYPE]); + + /* file length*/ + s->file_len = (tar2u64(&header[OFFS_LENGTH]) + (SECTOR_SIZE - 1)) & + ~(SECTOR_SIZE - 1); + s->file_sec = (header_offs / SECTOR_SIZE) + 1; + + header_offs += s->file_len + SECTOR_SIZE; + } while(!is_target_file(bs, header)); + + /* We found an image! */ + + if (header[OFFS_TYPE] == 'S') { + uint8_t isextended; + int i; + + for (i = OFFS_S_SP; i < (OFFS_S_SP + (4 * 24)); i += 24) + tar_sparse(s, tar2u64(&header[i]), tar2u64(&header[i+12])); + + s->file_len = tar2u64(&header[OFFS_S_LENGTH]); + isextended = header[OFFS_S_EXT]; + + while (isextended) { + if (bdrv_pread(s->hd, s->file_sec * SECTOR_SIZE, header, + SECTOR_SIZE) != SECTOR_SIZE) + goto fail; + + for (i = 0; i < (21 * 24); i += 24) + tar_sparse(s, tar2u64(&header[i]), tar2u64(&header[i+12])); + isextended = header[OFFS_SX_EXT]; + s->file_sec++; + } + tar_sparse(s, s->file_len, 1); + } + + return 0; + +fail: + fprintf(stderr, "Tar: Error opening file\n"); + bdrv_delete(s->hd); + return -EINVAL; +} + +typedef struct TarAIOCB { + BlockDriverAIOCB common; + QEMUBH *bh; +} TarAIOCB; + +static AIOPool tar_aio_pool = { + .aiocb_size = sizeof(TarAIOCB), +}; + +/* This callback gets invoked when we have pure sparseness */ +static void tar_sparse_cb(void *opaque) +{ + TarAIOCB *acb = (TarAIOCB *)opaque; + + acb->common.cb(acb->common.opaque, 0); + qemu_bh_delete(acb->bh); + qemu_aio_release(acb); +} + +/* This is where we get a request from a caller to read something */ +static BlockDriverAIOCB *tar_aio_readv(BlockDriverState *bs, + int64_t sector_num, QEMUIOVector *qiov, int nb_sectors, + BlockDriverCompletionFunc *cb, void *opaque) +{ + BDRVTarState *s = bs->opaque; + SparseCache *sparse; + int64_t sec_file = sector_num + s->file_sec; + int64_t start = sector_num * SECTOR_SIZE; + int64_t end = start + (nb_sectors * SECTOR_SIZE); + int i; + TarAIOCB *acb; + + for (i = 0; i < s->sparse_num; i++) { + sparse = &s->sparse[i]; + if (sparse->start > end) { + /* We expect the cache to be start increasing */ + break; + } else if ((sparse->start < start) && (sparse->end <= start)) { + /* sparse before our offset */ + sec_file -= (sparse->end - sparse->start) / SECTOR_SIZE; + } else if ((sparse->start <= start) && (sparse->end >= end)) { + /* all our sectors are sparse */ + char *buf = qemu_mallocz(nb_sectors * SECTOR_SIZE); + + acb = qemu_aio_get(&tar_aio_pool, bs, cb, opaque); + qemu_iovec_from_buffer(qiov, buf, nb_sectors * SECTOR_SIZE); + qemu_free(buf); + acb->bh = qemu_bh_new(tar_sparse_cb, acb); + qemu_bh_schedule(acb->bh); + + return &acb->common; + } else if (((sparse->start >= start) && (sparse->start < end)) || + ((sparse->end >= start) && (sparse->end < end))) { + /* we're semi-sparse (worst case) */ + /* let's go synchronous and read all sectors individually */ + char *buf = qemu_malloc(nb_sectors * SECTOR_SIZE); + uint64_t offs; + + for (offs = 0; offs < (nb_sectors * SECTOR_SIZE); + offs += SECTOR_SIZE) { + bdrv_pread(bs, (sector_num * SECTOR_SIZE) + offs, + buf + offs, SECTOR_SIZE); + } + + qemu_iovec_from_buffer(qiov, buf, nb_sectors * SECTOR_SIZE); + acb = qemu_aio_get(&tar_aio_pool, bs, cb, opaque); + acb->bh = qemu_bh_new(tar_sparse_cb, acb); + qemu_bh_schedule(acb->bh); + + return &acb->common; + } + } + + return bdrv_aio_readv(s->hd, sec_file, qiov, nb_sectors, + cb, opaque); +} + +static void tar_close(BlockDriverState *bs) +{ + dprintf("Close\n"); +} + +static int64_t tar_getlength(BlockDriverState *bs) +{ + BDRVTarState *s = bs->opaque; + dprintf("getlength -> %ld\n", s->file_len); + return s->file_len; +} + +static BlockDriver bdrv_tar = { + .format_name = "tar", + + .instance_size = sizeof(BDRVTarState), + .bdrv_open = tar_open, + .bdrv_close = tar_close, + .bdrv_getlength = tar_getlength, + .bdrv_probe = tar_probe, + + .bdrv_aio_readv = tar_aio_readv, +}; + +static void tar_block_init(void) +{ + bdrv_register(&bdrv_tar); +} + +block_init(tar_block_init);