From patchwork Wed Nov 1 15:43:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Nefedov X-Patchwork-Id: 833016 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="DXDwLfYO"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yRsxP2lPYz9sNw for ; Thu, 2 Nov 2017 02:45:29 +1100 (AEDT) Received: from localhost ([::1]:56371 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e9vCp-00084E-EJ for incoming@patchwork.ozlabs.org; Wed, 01 Nov 2017 11:45:27 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60387) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e9vC5-00081M-En for qemu-devel@nongnu.org; Wed, 01 Nov 2017 11:44:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e9vC0-0006a2-HQ for qemu-devel@nongnu.org; Wed, 01 Nov 2017 11:44:41 -0400 Received: from mail-eopbgr40116.outbound.protection.outlook.com ([40.107.4.116]:2657 helo=EUR03-DB5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e9vC0-0006Wl-0H; Wed, 01 Nov 2017 11:44:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=57dHvdkDyEs63GrcWEjnt1kfbbPodEqK46W9xNUr+Yc=; b=DXDwLfYOKm+tXTnIcKFRcff0kaQToetJoF1Zzh103uT2gpFMG8UgcIkBWBDykburP/7CsSOEiYLiHrXCSysJh+I7tO5xC/1MznwgD3WcqLr/PoqDG8Pa9yoSdyDFpCpnz74DEgNEYkeCGFkKoVG7up/q9UC+/P3Hze+qNyvxnlE= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=anton.nefedov@virtuozzo.com; Received: from xantnef-ws.sw.ru (195.214.232.6) by AM5PR0801MB1987.eurprd08.prod.outlook.com (2603:10a6:203:4b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Wed, 1 Nov 2017 15:44:31 +0000 From: Anton Nefedov To: qemu-devel@nongnu.org Date: Wed, 1 Nov 2017 18:43:53 +0300 Message-Id: <1509551048-129830-1-git-send-email-anton.nefedov@virtuozzo.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [195.214.232.6] X-ClientProxiedBy: VI1PR0501CA0017.eurprd05.prod.outlook.com (2603:10a6:800:92::27) To AM5PR0801MB1987.eurprd08.prod.outlook.com (2603:10a6:203:4b::14) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 549f3813-4a32-4a2b-3e2b-08d5213f721b X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(2017052603199); SRVR:AM5PR0801MB1987; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0801MB1987; 3:kGE7ao6JHu3NRc4UAtIRMVjZX2BgCLkaQjGpN0pC6SqHUw8A+DgQotpNCUTQKlvbEfyGTQjSkm8nBQi8Y1667/hL1H/sW5ynUzlLAV10QKqvk0y+idVhDu9cL55zqumM6u6TYSweqcbiRlLuqpCH55TX4elLikr+D3BWKrhfKtQl/rltq7y3EMNQsNqxFkzpRvspfl8QTVvbYaHDgGPUFhPQ6OY5VrDsMrbMGzOiXHo6GPnESSsd1//X/AYeWonA; 25:NtwEuh8HJTE7ZwSgoXDFumZwdXgcKl79LD+AxcGrn/a7/u4ANVpr+1Xd6YtPkuRg7FtFgDhjpqwKBZ66W4qpu3Ddkq2YqLH5Dd/u8CU2cLO1StcRkSBnYZiUrQuW6YhscXv1Ad4c1eP6PKrUWjjMVD5HY7naTLhz+L/3ESo6FQk41Lkj1sINbUqJrlLzPEVS0jCDgq54H/bcTtKu7DfgBa2mj3iQypbepZyqqX8yy32lJHjJIv9qYCFXroroyAn2ErgCq/+5PpCRQZdU33NJ1yaXhFagD86FavPFP7E23JvsQIulZOPHUl59+4zBhXDJgFIfOrq7IyZt0eBE3lrgJA==; 31:G3jELJXDkA+UXLNt1UTrZxOqrjBaVxBacbVQDypj/DC6vJJVw2O2e8jA2LU0Vkp7NeZaokLBRak1dYp5QugDqoM/P69wLQ6MRimyK0y3HhExy1MzdrmBoEQOSeXAkjLzbkaXswvftGp2dcFWt+Xlh9ZmHdM/zJ4hQQIm9hxrRLJS7yf3wP5TMQzRyfFPqn3G6BHrNqUv40CkHzOz08g0Kv/6pRYmpuKvkBigdaTS3AE= X-MS-TrafficTypeDiagnostic: AM5PR0801MB1987: X-Microsoft-Exchange-Diagnostics: 1; AM5PR0801MB1987; 20:7DUGIotsPWn2JgIdtmc1fYxUcEJFFg+bQK2Tje+xPTSeArNDdqGg3agjgGbexqgvo5g7lfI0pSAimLKOQJdHARItVwrc3FJcvw505N9H4shhKCk4lpISWQjQtQ/Y9zgrC7zfJjW4+W3H011C04fZ0Z+1hLHMu1k4zjM4JigZQ4U4W+J4OQwd4qmlKQqbzZqa3LRMm0Vb5oXtr2ctzneFi+v7bWqRmB+CVqx+2GP69aUkFi+hScwVXQCtovABo+o6rZjei+AET2p92Hh+1uj77ZfFdUezl0l6WoI5O1dfEhIW+Yx8b/l6omSp5HCkPh3It9JV9ZXfAwNL3fugDc8Sk5BQjN2QDNbIobKjdpxXBG3ejndZgDuPlx5oHxOyDgTEQ/ANvYW11mQtVWq/ZGu3vBQhxl3gOZxUiVAjfL/0Tk4=; 4:vimey7V0OSUr6aV3CSK7Taxo3vCJNY5Jt+ScfQnoR/PxOCbH4TJb4xkkpuAvRUqlukwUS5bWlWgevU5nE7VpiQTJ4SxjwM1YzWuO7OkOPGAJ0Tva4wNxv1x7D3v5ODqWU3xCeqelrnO38kT0xZVDvssMqdBgDZu/y0aWG5PXFXKV1WaXDodwcxGYmerVWJIk6xN2Nm8yWh+beBQIpCVl2EuboLgAMM4Ytz1tVz8ba4eHVihUFbiufBH30zt+4O/1DLKAcmIv7b4o2Vv8Ja5YoA== X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231020)(100000703101)(100105400095)(3002001)(6041248)(20161123555025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123564025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM5PR0801MB1987; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM5PR0801MB1987; X-Forefront-PRVS: 0478C23FE0 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(6009001)(376002)(346002)(189002)(199003)(6666003)(16526018)(53936002)(107886003)(48376002)(86362001)(6916009)(3846002)(6116002)(101416001)(50986999)(106356001)(69596002)(5660300001)(25786009)(105586002)(316002)(53416004)(2351001)(16586007)(2361001)(33646002)(4326008)(5003940100001)(478600001)(966005)(66066001)(36756003)(50226002)(6306002)(6512007)(47776003)(2906002)(8936002)(6506006)(97736004)(6486002)(68736007)(305945005)(50466002)(81166006)(81156014)(8676002)(189998001)(7736002); DIR:OUT; SFP:1102; SCL:1; SRVR:AM5PR0801MB1987; H:xantnef-ws.sw.ru; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: virtuozzo.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0801MB1987; 23:KIqL4yOflmmlBdBDq/n3JjmfmCKCTOfEFwGf6bc?= +8TpBtnLakDlG0tChh2JDpbP0YgppuOzbuHXalYRDgICm/6aCiorcAOBQ+qGRaXA7xawJAn2zL7xx/6raH8FS+fQbcqo+LCtWhwPjqgaadZqdXTJkXRpACWHl571oC1jh2nTWomV/P31BC5fN9lmc1FDc1hYtqYym9ulowlkPKc3VdLc8vXcQscVxjkLG8vfZmz+5z9HOuKCV8igf6ikvdSXLRNT40C42Vacn9EJYCqClUkwboezDY2wB6zMOeTsJfqUSJ7K/9xEY12SOEgzi6yaAbmYiHiIoYBk1Dt3g8hPOXOM6m9gMqMq/9q97nsgORR8rF3FxLjTUOEG6uL21PVV6XBIEyTl6bpr2QkNKxWIhPqasFsnzE14WevT/rRft2KDBT7YgGrOjnjDRHurI/GO5xDUHLI2dy+s+QRnxgPhe/WDdjwb3HVcCprgZc0hJAv+vV4XlaS9wYLj3k141apdXw11Xq/Hffr9oaxpFAKJ8w8Lg3GnV0qaqGl+S9r0I6SHqfv51lLQA4WIT9oZYVYuI0hA9/38m2wNTAf7I9Ouyj2d4Tjits1aKA48St4QPXMq+EvETDbSRKNI5HB9by3cbkj41NqwQdX4I2FQ1BS9aGV4n/lmruRp6BQsAoOJqSgeO4FmN8De01IJT/YD4RP4SNh0qhd1aEji2uvG/4TXQvoWDk6BvaZ5cEgxyM+vS6S5sRvGlyEnaJLzhxgDLshWtlfbtmr9qDlrzi1ummFvmX2jjZstjVouNRzJgJTr+o1SeSh42QgSZi/P+GM+kvPTBiWY47RJl3PTkDIZB2vkAHNcLq112ZkH30NHGwm3nfUka1sbnn2T0JAuCTCxyRfBfPmu+gcmpnUekBfN9MTTTROuNzVVvVHVa5gcD6FhBVA2Yo829h/NJhF4K9mtAKRpfPEY9ezE3rX21Q45r5830Xex2jiBiCu6cpuIp9bJ+G6sxrHMgfLtw8L/6s09j+eyu16sf2tZYZN6QBDwac+jgqZcQn8ZvOe0KPBBPhKCge7fjHG3/9KvF8OhEU2BzOUgw9jn/2V+401wEQOCgQLxyypGcU0ijiGF14VJXTHI9tvSyuxdeHNkwgychCsMHR3Pfl5ZgWwbZ5QSHX7/vKOLPHg== X-Microsoft-Exchange-Diagnostics: 1; AM5PR0801MB1987; 6:toxZQP/R0Wduf7EL0bwRHJgMbFybo8XQpcLA4gGTM8T2MP68WDGBWhIPW/75i6xLSKTI+NDkYpodkUSYdGJUEXEqp9wj4OwCGA4nyEkCQF2M3QoYIautoiNr2r/L39UY6H98ZmUBaVz1pYD/zbeUvV7x0+/yLZXaPjjFwBxDDcTGaZtGfSX/P2bsYrVDWNUU/z+OYZFz1hU6ZGG4WkDSbDcFRyf4rzGctI1KH00onQWHFuGFiEs3zZ39LyCCaxcP9HgrtgEfuw17iEOtgu+t4IGEjlZ15Wk6hviWa3oxlbmSou7EhLpwrCtjLJKAqMhNJirZSmkgvfvwZf8Q6biWQt3pav0p68MLPq59uuFCoXc=; 5:G3JaLmalP8ItNv/BZVjbnqr0O4aVPwuWNBDW1fQ09zuUtnndZC3ZMqhL7Z5Fr/ZhZkNE2pmWXIqf5kuJM8ETeraqPLxy2z9kYIxH6k149FifezIfkdF4Rnnmf/cYkT1psDZFrxfBo1BJz7JamSPocZRpkg4lyvEEb3YIBqCrttA=; 24:sywv1Xm3i8xhZO4ZZcu00iokK9QcDlu52Mms715hBZObP0D8T9IQCRCr9woWWqXQBp4Kqw2JcoJF2c37uX/uXm9lh1eiWf4plUE3v7Un1vg=; 7:nt2LpGNdjbgYNSwmLoHdXTwVK7C6DYsbBUAe/9irISk4XqcdB12IfbfSvoUOCvW6E9/uJW5zimeGDH2BFrDWiLV7aaAmgmv8eaQmxHaF12va2jUoxfkAuSTJvNW7utMbDHEfZroN4Ff62OOpxtd22U9C3BzBTWXvQHydTRJV27LW0I8PzZ5XsXOe8TTNemSt9BweBLrh7niWnfumprGNnlLRhANjwqjo1HVJKKHbgdU/wN2lyHdbPuBst2y+gFT+ SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM5PR0801MB1987; 20:j9BW5qt/IxpbDPhmy1XkIvPdFfF+IPgyrv/AZGon3uXVg5KE8BoAGeQO2JzXc6IGKfbXBrntjXyZC9AOsTSXnhUWuWqZHgEj0pH8bMGIp85tVvqFZPKX4OKlwGXvLoRCyy7PAZafVF4YZIRtstWxZead2RpORikz2npMmsyR/fY= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2017 15:44:31.5399 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 549f3813-4a32-4a2b-3e2b-08d5213f721b X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0801MB1987 X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 [fuzzy] X-Received-From: 40.107.4.116 Subject: [Qemu-devel] [PATCH v5 00/15] qcow2: space preallocation and COW improvements X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Anton Nefedov , berto@igalia.com, den@virtuozzo.com, qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" v5: rebased, patches slightly reordered; cluster allocation (to avoid writing zero-buffers) now goes before an optional ahead preallocation v4: http://lists.nongnu.org/archive/html/qemu-devel/2017-08/msg00109.html This pull request is to improve a few performance problems of qcow2 format: 1. non cluster-aligned write requests (to unallocated clusters) explicitly pad data with zeroes if there is no backing data. Resulting increase in ops number and potential cluster fragmentation (on the host file) is already solved by: ee22a9d qcow2: Merge the writing of the COW regions with the guest data However, in case of zero COW regions, that can be avoided at all but the whole clusters are preallocated and zeroed in a single efficient write_zeroes() operation 2. moreover, efficient write_zeroes() operation can be used to preallocate space megabytes (*configurable number) ahead which gives noticeable improvement on some storage types (e.g. distributed storage) where the space allocation operation might be expensive) 3. this will also allow to enable simultaneous writes to the same unallocated cluster after the space has been allocated & zeroed but before the first data is written and the cluster is linked to L2. (Not included in this patchset). Efficient write_zeroes usually implies that the blocks are not actually written to but only reserved and marked as zeroed by the storage. In this patchset, file-posix driver is marked as supporting this operation if it supports (/configured to support) fallocate() operation. Existing bdrv_write_zeroes() falls back to writing zero buffers if write_zeroes is not supported by the driver. A new flag (BDRV_REQ_ALLOCATE) is introduced to avoid that but return ENOTSUP. Such allocate requests are also implemented to possibly overlap with the other requests. No wait is performed but an error returned in such case as well. So the operation should be considered advisory and a fallback scenario still handled by the caller (in this case, qcow2 driver). Anton Nefedov (12): mirror: inherit supported write/zero flags blkverify: set supported write/zero flags block: introduce BDRV_REQ_ALLOCATE flag block: treat BDRV_REQ_ALLOCATE as serialising file-posix: support BDRV_REQ_ALLOCATE block: support BDRV_REQ_ALLOCATE in passthrough drivers qcow2: move is_zero() up qcow2: skip writing zero buffers to empty COW areas qcow2: set inactive flag qcow2: do not zero out clusters if already preallocated iotest 198: test BDRV_REQ_ALLOCATE iotest 134: test cluster-misaligned encrypted write Denis V. Lunev (2): qcow2: preallocation at image expand qcow2: truncate preallocated space Pavel Butsykin (1): qcow2: check space leak at the end of the image block/qcow2.h | 18 ++++ include/block/block.h | 6 +- include/block/block_int.h | 2 +- block/blkdebug.c | 3 +- block/blkverify.c | 9 ++ block/file-posix.c | 8 ++ block/io.c | 47 +++++++-- block/mirror.c | 5 + block/qcow2-cluster.c | 14 ++- block/qcow2-refcount.c | 7 ++ block/qcow2.c | 202 +++++++++++++++++++++++++++++++++---- block/raw-format.c | 3 +- block/trace-events | 1 + qemu-options.hx | 4 + tests/qemu-iotests/026.out | 94 +++++++++++++---- tests/qemu-iotests/026.out.nocache | 94 +++++++++++++---- tests/qemu-iotests/029.out | 5 +- tests/qemu-iotests/060 | 2 +- tests/qemu-iotests/060.out | 13 ++- tests/qemu-iotests/061.out | 5 +- tests/qemu-iotests/066 | 2 +- tests/qemu-iotests/066.out | 9 +- tests/qemu-iotests/098.out | 7 +- tests/qemu-iotests/108.out | 5 +- tests/qemu-iotests/112.out | 5 +- tests/qemu-iotests/134 | 9 ++ tests/qemu-iotests/134.out | 10 ++ tests/qemu-iotests/198 | 146 +++++++++++++++++++++++++++ tests/qemu-iotests/198.out | 50 +++++++++ tests/qemu-iotests/group | 1 + 30 files changed, 693 insertions(+), 93 deletions(-) create mode 100755 tests/qemu-iotests/198 create mode 100644 tests/qemu-iotests/198.out