From patchwork Fri Oct 20 21:56:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852842 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=inFcDJSP; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz3D0MJVz23jq for ; Sat, 21 Oct 2023 08:57:43 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTy-0003rO-1i; Fri, 20 Oct 2023 17:56:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTu-0003pO-AW; Fri, 20 Oct 2023 17:56:35 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008G5-Mz; Fri, 20 Oct 2023 17:56:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=iHfVvyVjq4g5YNZ0DhXc240zSGDlc6cdMexUlDF6B1U=; b=inFcDJSPdY6O yEuiW+4CnlCEsWiwFYN31xijQPeJ9saVyjqYdoIPW1F940fSNmRGoDBt8bb9CANQK6dh0x7N685KY fwc36Q5AId6jcjqiMh5irTzH9GebcUutkjNC3XhXjKm9QvPeGZQ3I/Fadf6Ik80Korslllhzte0jG pSDC0+iJjFx4lOQqVuqhH0hww1/N4xLoTD0F8yT1mgiUKCo61uK6aC3JE2yInANHVvKOgThCUgw2+ P4EFoio/P5riIunsvANW3Ut8n6aTOwz59cCUqVliro28I760cQFnWKlPL6gT+7SX9zMHmwWaNJ5nX vVWPbQl7zcfowYpBaI1tgg==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-1k; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 1/7] qcow2: make function update_refcount_discard() global Date: Sat, 21 Oct 2023 00:56:16 +0300 Message-Id: <20231020215622.789260-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We are going to need it for discarding separate subclusters. The function itself doesn't do anything with the refcount tables, it simply adds a discard request to the queue, so rename it to qcow2_queue_discard(). Signed-off-by: Andrey Drobyshev Reviewed-by: Hanna Czenczek --- block/qcow2-refcount.c | 8 ++++---- block/qcow2.h | 2 ++ 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 0266542cee..2026f5fa21 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -754,8 +754,8 @@ void qcow2_process_discards(BlockDriverState *bs, int ret) } } -static void update_refcount_discard(BlockDriverState *bs, - uint64_t offset, uint64_t length) +void qcow2_queue_discard(BlockDriverState *bs, uint64_t offset, + uint64_t length) { BDRVQcow2State *s = bs->opaque; Qcow2DiscardRegion *d, *p, *next; @@ -902,7 +902,7 @@ update_refcount(BlockDriverState *bs, int64_t offset, int64_t length, } if (s->discard_passthrough[type]) { - update_refcount_discard(bs, cluster_offset, s->cluster_size); + qcow2_queue_discard(bs, cluster_offset, s->cluster_size); } } } @@ -3619,7 +3619,7 @@ qcow2_discard_refcount_block(BlockDriverState *bs, uint64_t discard_block_offs) /* discard refblock from the cache if refblock is cached */ qcow2_cache_discard(s->refcount_block_cache, refblock); } - update_refcount_discard(bs, discard_block_offs, s->cluster_size); + qcow2_queue_discard(bs, discard_block_offs, s->cluster_size); return 0; } diff --git a/block/qcow2.h b/block/qcow2.h index 29958c512b..75d6a1b61b 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -891,6 +891,8 @@ int coroutine_fn qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *re BdrvCheckMode fix); void qcow2_process_discards(BlockDriverState *bs, int ret); +void qcow2_queue_discard(BlockDriverState *bs, uint64_t offset, + uint64_t length); int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size); From patchwork Fri Oct 20 21:56:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852844 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=dHxm6Tce; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz3m08sVz23jq for ; Sat, 21 Oct 2023 08:58:12 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTz-0003sc-BE; Fri, 20 Oct 2023 17:56:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTu-0003pM-A6; Fri, 20 Oct 2023 17:56:35 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008Fz-NS; Fri, 20 Oct 2023 17:56:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=5EW9dBv700hG96A7F9K7bifxv268XKy4vOdbQChMeHA=; b=dHxm6Tcexdsh KUVCDk/2TURH/Z+724Ngp6cA7fP6RjGELjEYv5jb9Nn8jFK0xsuXQ8JkiZOVOHmKFIw26BrQrcE1E sr+e+C58RAIi6/hq1vb3nSgedHstWXE6gTLFPc4TBwOy2xHxkq4NHf/Tmh+IFMksoRu/XZ9n53X1W juMWNLq3DZWzHtWJOs9cG5R2s+xJXlQR728nvqbJht8Opb3pCVsqXFQQ28LE7bKFYhh3Nze/Zc09r qTdNagMd/K28u/Z4XO1+BHQY/wEoczX8HHs96K+LIskKJRlfP7+ANxReOJSgUUhyEKJtwLbeYxVHT Yb3aTYCJTqAd/DAaSHa9sw==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-1v; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 2/7] qcow2: add get_sc_range_info() helper for working with subcluster ranges Date: Sat, 21 Oct 2023 00:56:17 +0300 Message-Id: <20231020215622.789260-3-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This helper simply obtains the l2 table parameters of the cluster which contains the given subclusters range. Right now this info is being obtained and used by zero_l2_subclusters(). As we're about to introduce the subclusters discard operation, this helper would let us avoid code duplication. Also introduce struct SubClusterRangeInfo, which would contain all the needed params. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 90 +++++++++++++++++++++++++++++-------------- 1 file changed, 62 insertions(+), 28 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 904f00d1b3..8801856b93 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -32,6 +32,13 @@ #include "qemu/memalign.h" #include "trace.h" +typedef struct SubClusterRangeInfo { + uint64_t *l2_slice; + int l2_index; + uint64_t l2_entry; + uint64_t l2_bitmap; +} SubClusterRangeInfo; + int coroutine_fn qcow2_shrink_l1_table(BlockDriverState *bs, uint64_t exact_size) { @@ -1892,6 +1899,50 @@ again: return 0; } +static int get_sc_range_info(BlockDriverState *bs, uint64_t offset, + unsigned nb_subclusters, + SubClusterRangeInfo *scri) +{ + BDRVQcow2State *s = bs->opaque; + int ret, sc_cleared = 0, sc_index = offset_to_sc_index(s, offset); + QCow2SubclusterType sctype; + + /* Here we only work with the subclusters within single cluster. */ + assert(nb_subclusters > 0 && nb_subclusters < s->subclusters_per_cluster); + assert(sc_index + nb_subclusters <= s->subclusters_per_cluster); + assert(offset_into_subcluster(s, offset) == 0); + + ret = get_cluster_table(bs, offset, &scri->l2_slice, &scri->l2_index); + if (ret < 0) { + return ret; + } + + scri->l2_entry = get_l2_entry(s, scri->l2_slice, scri->l2_index); + scri->l2_bitmap = get_l2_bitmap(s, scri->l2_slice, scri->l2_index); + + do { + qcow2_get_subcluster_range_type(bs, scri->l2_entry, scri->l2_bitmap, + sc_index, &sctype); + if (ret < 0) { + return ret; + } + + switch (sctype) { + case QCOW2_SUBCLUSTER_COMPRESSED: + /* We cannot partially zeroize/discard compressed clusters. */ + return -ENOTSUP; + case QCOW2_SUBCLUSTER_INVALID: + return -EINVAL; + default: + break; + } + + sc_cleared += ret; + } while (sc_cleared < nb_subclusters); + + return 0; +} + /* * This discards as many clusters of nb_clusters as possible at once (i.e. * all clusters in the same L2 slice) and returns the number of discarded @@ -2097,44 +2148,27 @@ zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, unsigned nb_subclusters) { BDRVQcow2State *s = bs->opaque; - uint64_t *l2_slice; - uint64_t old_l2_bitmap, l2_bitmap; - int l2_index, ret, sc = offset_to_sc_index(s, offset); + uint64_t new_l2_bitmap; + int ret, sc = offset_to_sc_index(s, offset); + SubClusterRangeInfo scri = { 0 }; - /* For full clusters use zero_in_l2_slice() instead */ - assert(nb_subclusters > 0 && nb_subclusters < s->subclusters_per_cluster); - assert(sc + nb_subclusters <= s->subclusters_per_cluster); - assert(offset_into_subcluster(s, offset) == 0); - - ret = get_cluster_table(bs, offset, &l2_slice, &l2_index); + ret = get_sc_range_info(bs, offset, nb_subclusters, &scri); if (ret < 0) { - return ret; - } - - switch (qcow2_get_cluster_type(bs, get_l2_entry(s, l2_slice, l2_index))) { - case QCOW2_CLUSTER_COMPRESSED: - ret = -ENOTSUP; /* We cannot partially zeroize compressed clusters */ goto out; - case QCOW2_CLUSTER_NORMAL: - case QCOW2_CLUSTER_UNALLOCATED: - break; - default: - g_assert_not_reached(); } - old_l2_bitmap = l2_bitmap = get_l2_bitmap(s, l2_slice, l2_index); - - l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); - l2_bitmap &= ~QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); + new_l2_bitmap = scri.l2_bitmap; + new_l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); + new_l2_bitmap &= ~QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); - if (old_l2_bitmap != l2_bitmap) { - set_l2_bitmap(s, l2_slice, l2_index, l2_bitmap); - qcow2_cache_entry_mark_dirty(s->l2_table_cache, l2_slice); + if (new_l2_bitmap != scri.l2_bitmap) { + set_l2_bitmap(s, scri.l2_slice, scri.l2_index, new_l2_bitmap); + qcow2_cache_entry_mark_dirty(s->l2_table_cache, scri.l2_slice); } ret = 0; out: - qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); + qcow2_cache_put(s->l2_table_cache, (void **) &scri.l2_slice); return ret; } From patchwork Fri Oct 20 21:56:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852845 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=Spc48Bus; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz3t367tz23jq for ; Sat, 21 Oct 2023 08:58:18 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTv-0003pR-9m; Fri, 20 Oct 2023 17:56:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTs-0003nm-RZ; Fri, 20 Oct 2023 17:56:32 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008G0-Iw; Fri, 20 Oct 2023 17:56:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=AAm3BCi7oLK+rMtTaRnf4YstYpBZyRvRpC4m7z5RCrM=; b=Spc48BusaTgy M2MveG7EpevasYorRVBbYmuwedD8zP1/iGbmEnZwwZcXVtgSJp/o70vPjv4fZ/mtojLoy1xWvdZiw fpTBjmscgIIu9mW+inf9EbmSc0GUNXHycQl31xCMQuTIWGowuzsY1g+IKPooWrxPRpr/mHsXne2xo h5pUJOLhfey1IOW0zFErn4feXRZbEZRre63H9sG7c+q1kUL4J7/x6hX5VVZCipmi3ulbraD8g5gFp PHi6KGc/pA6DgsYFCTxWq7TLpgQ2kXtBZrJoYea4V594TePJUTlCl0blRd/hRd+c28wa8jlIVrJiR W/J280PSxSPN4dwO7HCopw==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-26; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 3/7] qcow2: zeroize the entire cluster when there're no non-zero subclusters Date: Sat, 21 Oct 2023 00:56:18 +0300 Message-Id: <20231020215622.789260-4-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org When zeroizing the last non-zero subclusters within single cluster, it makes sense to go zeroize the entire cluster and go down zero_in_l2_slice() path right away. That way we'd also update the corresponding refcount table. Signed-off-by: Andrey Drobyshev Reviewed-by: Hanna Czenczek --- block/qcow2-cluster.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 8801856b93..7c6fa5524c 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -2145,7 +2145,7 @@ zero_in_l2_slice(BlockDriverState *bs, uint64_t offset, static int coroutine_fn GRAPH_RDLOCK zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, - unsigned nb_subclusters) + unsigned nb_subclusters, int flags) { BDRVQcow2State *s = bs->opaque; uint64_t new_l2_bitmap; @@ -2161,6 +2161,17 @@ zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, new_l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); new_l2_bitmap &= ~QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); + /* + * If there're no non-zero subclusters left, we might as well zeroize + * the entire cluster. That way we'd also update the refcount table. + */ + if ((new_l2_bitmap & QCOW_L2_BITMAP_ALL_ZEROES) == + QCOW_L2_BITMAP_ALL_ZEROES) { + qcow2_cache_put(s->l2_table_cache, (void **) &scri.l2_slice); + return zero_in_l2_slice(bs, QEMU_ALIGN_DOWN(offset, s->cluster_size), + 1, flags); + } + if (new_l2_bitmap != scri.l2_bitmap) { set_l2_bitmap(s, scri.l2_slice, scri.l2_index, new_l2_bitmap); qcow2_cache_entry_mark_dirty(s->l2_table_cache, scri.l2_slice); @@ -2221,7 +2232,7 @@ int coroutine_fn qcow2_subcluster_zeroize(BlockDriverState *bs, uint64_t offset, if (head) { ret = zero_l2_subclusters(bs, offset - head, - size_to_subclusters(s, head)); + size_to_subclusters(s, head), flags); if (ret < 0) { goto fail; } @@ -2242,7 +2253,8 @@ int coroutine_fn qcow2_subcluster_zeroize(BlockDriverState *bs, uint64_t offset, } if (tail) { - ret = zero_l2_subclusters(bs, end_offset, size_to_subclusters(s, tail)); + ret = zero_l2_subclusters(bs, end_offset, + size_to_subclusters(s, tail), flags); if (ret < 0) { goto fail; } From patchwork Fri Oct 20 21:56:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852843 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=RWsp3xhU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz3D0Qknz23kJ for ; Sat, 21 Oct 2023 08:57:43 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxU8-0003xp-E7; Fri, 20 Oct 2023 17:56:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTy-0003s4-E5; Fri, 20 Oct 2023 17:56:39 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008Fy-M4; Fri, 20 Oct 2023 17:56:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=AOdVB5L+10G1WQ+8o75fKXSttdivBvesl0iH9zSiDRA=; b=RWsp3xhUFTQ3 kVAhfThRyMtF2JWs4amcOI/U/r68AliLrhp+eNS2WWZ+WGI7xIzAUyFy96Tn4SwyDSa0XcnZjD3we 97oaB/FcUDRkBaxM8eFJOPrUFB5yWH1rUsdJ/BlrqmXQytOWeES2uQFyqemCUw+sALC/sYl0BY4p8 LDVMXnQyGh1fnkCNx6RZ50sM+tWkQNrZsyXBxkbCC3G8yjG+prPy5ox9V+Aei2H8vlgw3mjrDkuBa QSQWXiLwQSjut9d5lUkKPlb3078rKKK2hSloGhc8d9yg2xvA4fP+ZKEDt0DBeae1QMCvj2EaQ+8Dp OR3qf2KJ9skjJ/npaXAetQ==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-2H; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 4/7] qcow2: make subclusters discardable Date: Sat, 21 Oct 2023 00:56:19 +0300 Message-Id: <20231020215622.789260-5-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This commit makes the discard operation work on the subcluster level rather than cluster level. It introduces discard_l2_subclusters() function and makes use of it in qcow2 discard implementation, much like it's done with zero_in_l2_slice() / zero_l2_subclusters(). It also changes the qcow2 driver pdiscard_alignment to subcluster_size. That way subcluster-aligned discards lead to actual fallocate(PUNCH_HOLE) operation and free host disk space. This feature will let us gain additional disk space on guest TRIM/discard requests, especially when using large enough clusters (1M, 2M) with subclusters enabled. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 100 ++++++++++++++++++++++++++++++++++++++++-- block/qcow2.c | 8 ++-- 2 files changed, 101 insertions(+), 7 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 7c6fa5524c..cf40f2dc12 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -2042,6 +2042,74 @@ discard_in_l2_slice(BlockDriverState *bs, uint64_t offset, uint64_t nb_clusters, return nb_clusters; } +static int coroutine_fn GRAPH_RDLOCK +discard_l2_subclusters(BlockDriverState *bs, uint64_t offset, + uint64_t nb_subclusters, + enum qcow2_discard_type type, + bool full_discard, + SubClusterRangeInfo *pscri) +{ + BDRVQcow2State *s = bs->opaque; + uint64_t new_l2_bitmap, l2_bitmap_mask; + int ret, sc = offset_to_sc_index(s, offset); + SubClusterRangeInfo scri = { 0 }; + + if (!pscri) { + ret = get_sc_range_info(bs, offset, nb_subclusters, &scri); + if (ret < 0) { + goto out; + } + } else { + scri = *pscri; + } + + l2_bitmap_mask = QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); + new_l2_bitmap = scri.l2_bitmap; + new_l2_bitmap &= ~l2_bitmap_mask; + + /* + * If there're no allocated subclusters left, we might as well discard + * the entire cluster. That way we'd also update the refcount table. + */ + if (!(new_l2_bitmap & QCOW_L2_BITMAP_ALL_ALLOC)) { + return discard_in_l2_slice(bs, + QEMU_ALIGN_DOWN(offset, s->cluster_size), + 1, type, full_discard); + } + + /* + * Full discard means we fall through to the backing file, thus we only + * need to mark the subclusters as deallocated. + * + * Non-full discard means subclusters should be explicitly marked as + * zeroes. In this case QCOW2 specification requires the corresponding + * allocation status bits to be unset as well. If the subclusters are + * deallocated in the first place and there's no backing, the operation + * can be skipped. + */ + if (!full_discard && + (bs->backing || scri.l2_bitmap & l2_bitmap_mask)) { + new_l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); + } + + if (scri.l2_bitmap != new_l2_bitmap) { + set_l2_bitmap(s, scri.l2_slice, scri.l2_index, new_l2_bitmap); + qcow2_cache_entry_mark_dirty(s->l2_table_cache, scri.l2_slice); + } + + if (s->discard_passthrough[type]) { + qcow2_queue_discard(bs, (scri.l2_entry & L2E_OFFSET_MASK) + + offset_into_cluster(s, offset), + nb_subclusters * s->subcluster_size); + } + + ret = 0; +out: + qcow2_cache_put(s->l2_table_cache, (void **) &scri.l2_slice); + + return ret; +} + int qcow2_cluster_discard(BlockDriverState *bs, uint64_t offset, uint64_t bytes, enum qcow2_discard_type type, bool full_discard) @@ -2049,19 +2117,36 @@ int qcow2_cluster_discard(BlockDriverState *bs, uint64_t offset, BDRVQcow2State *s = bs->opaque; uint64_t end_offset = offset + bytes; uint64_t nb_clusters; + unsigned head, tail; int64_t cleared; int ret; /* Caller must pass aligned values, except at image end */ - assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); - assert(QEMU_IS_ALIGNED(end_offset, s->cluster_size) || + assert(QEMU_IS_ALIGNED(offset, s->subcluster_size)); + assert(QEMU_IS_ALIGNED(end_offset, s->subcluster_size) || end_offset == bs->total_sectors << BDRV_SECTOR_BITS); - nb_clusters = size_to_clusters(s, bytes); + head = MIN(end_offset, ROUND_UP(offset, s->cluster_size)) - offset; + offset += head; + + tail = (end_offset >= bs->total_sectors << BDRV_SECTOR_BITS) ? 0 : + end_offset - MAX(offset, start_of_cluster(s, end_offset)); + end_offset -= tail; s->cache_discards = true; + if (head) { + ret = discard_l2_subclusters(bs, offset - head, + size_to_subclusters(s, head), type, + full_discard, NULL); + if (ret < 0) { + goto fail; + } + } + /* Each L2 slice is handled by its own loop iteration */ + nb_clusters = size_to_clusters(s, end_offset - offset); + while (nb_clusters > 0) { cleared = discard_in_l2_slice(bs, offset, nb_clusters, type, full_discard); @@ -2074,6 +2159,15 @@ int qcow2_cluster_discard(BlockDriverState *bs, uint64_t offset, offset += (cleared * s->cluster_size); } + if (tail) { + ret = discard_l2_subclusters(bs, end_offset, + size_to_subclusters(s, tail), type, + full_discard, NULL); + if (ret < 0) { + goto fail; + } + } + ret = 0; fail: s->cache_discards = false; diff --git a/block/qcow2.c b/block/qcow2.c index aa01d9e7b5..66961fa59e 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1966,7 +1966,7 @@ static void qcow2_refresh_limits(BlockDriverState *bs, Error **errp) bs->bl.request_alignment = qcrypto_block_get_sector_size(s->crypto); } bs->bl.pwrite_zeroes_alignment = s->subcluster_size; - bs->bl.pdiscard_alignment = s->cluster_size; + bs->bl.pdiscard_alignment = s->subcluster_size; } static int GRAPH_UNLOCKED @@ -4102,11 +4102,11 @@ qcow2_co_pdiscard(BlockDriverState *bs, int64_t offset, int64_t bytes) return -ENOTSUP; } - if (!QEMU_IS_ALIGNED(offset | bytes, s->cluster_size)) { - assert(bytes < s->cluster_size); + if (!QEMU_IS_ALIGNED(offset | bytes, bs->bl.pdiscard_alignment)) { + assert(bytes < bs->bl.pdiscard_alignment); /* Ignore partial clusters, except for the special case of the * complete partial cluster at the end of an unaligned file */ - if (!QEMU_IS_ALIGNED(offset, s->cluster_size) || + if (!QEMU_IS_ALIGNED(offset, bs->bl.pdiscard_alignment) || offset + bytes != bs->total_sectors * BDRV_SECTOR_SIZE) { return -ENOTSUP; } From patchwork Fri Oct 20 21:56:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852841 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=bjCGOG30; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz3D0R1Dz23kW for ; Sat, 21 Oct 2023 08:57:42 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTz-0003sd-EP; Fri, 20 Oct 2023 17:56:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTu-0003pL-9Y; Fri, 20 Oct 2023 17:56:34 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008G1-PA; Fri, 20 Oct 2023 17:56:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=mEK1pV36MoJx+qzm6bIJ01vudQTCQDmFPR0gocEfWEI=; b=bjCGOG30fRcL jcRsgs+LdCPJ2Hf2WfPFw3sroGjwJykanMKSAc1/y7c/hOYELZ6Uy/OD75SgUNKfeM74NHAaMxNmS 4IYCPq/K+r40JpZ+6JwHVXl13vEZYB7pVv7qhxCJefePEAVU7OfvYPmcFfJsBLR4hLSLJimDm2J7d HoNjTOGFL2lQH487iCfYliS+NOQsUtLKJYRLEiSTcWx0omf6XTO3RSwGv+QF4pw/RNCui7BPZgWpF 2q1l5MAX8pl1dAsg6rsR2fQ5k/UtUkbhFHCVtyrh2Brx47YJPoQyaGQPViinQqueFWCT6O9jG4pRK lDZN0UCLDjo9d5DW29t/8w==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-2S; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 5/7] qcow2: zero_l2_subclusters: fall through to discard operation when requested Date: Sat, 21 Oct 2023 00:56:20 +0300 Message-Id: <20231020215622.789260-6-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org When zeroizing subclusters within single cluster, detect usage of the BDRV_REQ_MAY_UNMAP flag and fall through to the subcluster-based discard operation, much like it's done with the cluster-based discards. That way subcluster-aligned operations "qemu-io -c 'write -z -u ...'" will lead to actual unmap. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index cf40f2dc12..040251f2c3 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -2242,7 +2242,7 @@ zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, unsigned nb_subclusters, int flags) { BDRVQcow2State *s = bs->opaque; - uint64_t new_l2_bitmap; + uint64_t new_l2_bitmap, l2_bitmap_mask; int ret, sc = offset_to_sc_index(s, offset); SubClusterRangeInfo scri = { 0 }; @@ -2251,9 +2251,10 @@ zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, goto out; } + l2_bitmap_mask = QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); new_l2_bitmap = scri.l2_bitmap; - new_l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); - new_l2_bitmap &= ~QCOW_OFLAG_SUB_ALLOC_RANGE(sc, sc + nb_subclusters); + new_l2_bitmap |= QCOW_OFLAG_SUB_ZERO_RANGE(sc, sc + nb_subclusters); + new_l2_bitmap &= ~l2_bitmap_mask; /* * If there're no non-zero subclusters left, we might as well zeroize @@ -2266,6 +2267,16 @@ zero_l2_subclusters(BlockDriverState *bs, uint64_t offset, 1, flags); } + /* + * If the request allows discarding subclusters and they're actually + * allocated, we go down the discard path since after the discard + * operation the subclusters are going to be read as zeroes anyway. + */ + if ((flags & BDRV_REQ_MAY_UNMAP) && (scri.l2_bitmap & l2_bitmap_mask)) { + return discard_l2_subclusters(bs, offset, nb_subclusters, + QCOW2_DISCARD_REQUEST, false, &scri); + } + if (new_l2_bitmap != scri.l2_bitmap) { set_l2_bitmap(s, scri.l2_slice, scri.l2_index, new_l2_bitmap); qcow2_cache_entry_mark_dirty(s->l2_table_cache, scri.l2_slice); From patchwork Fri Oct 20 21:56:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852848 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=Ys4kzVyy; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz4B75Cgz23jq for ; Sat, 21 Oct 2023 08:58:34 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTv-0003qQ-LX; Fri, 20 Oct 2023 17:56:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTs-0003mU-EM; Fri, 20 Oct 2023 17:56:32 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008G3-NA; Fri, 20 Oct 2023 17:56:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=Xc0s5rzni+x47c5YoKWzgM1EXDpgmhujUFkXgIBX2Ec=; b=Ys4kzVyyw8Ap cHO1/oos/k/dlUAxWrB8wpwE5gT2fbCiawE1Upl9KOr2hW+MTJ5M3B/ilCOP40UfUhr2E9lok61+4 cXhi/1ReNLVfDS6ysR+HqKUgG3KSy2Tt9IGSPXKjnT1uV+9SRgyRU5yvQyQbleGIYUYAmYj1gYZi6 myocjmAJWTzATKJXBNJc+sMcw7Yr4bzR7nz+bt4tU7G2JR5CSBhwJn1YORKviMDyZdOGAnV/UEIb6 WzAcnP0QJC5Z4bK0Hfl/UStde/4DH5+ynEtfvznLt92XoTReD2GOslURBQFeKJSJ+uO4C6AyOpRFu 4yT35uXrG8HQ8kzr1OPW7Q==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-2d; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 6/7] iotests/common.rc: add disk_usage function Date: Sat, 21 Oct 2023 00:56:21 +0300 Message-Id: <20231020215622.789260-7-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Move the definition from iotests/250 to common.rc. This is used to detect real disk usage of sparse files. In particular, we want to use it for checking subclusters-based discards. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/250 | 5 ----- tests/qemu-iotests/common.rc | 6 ++++++ 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/250 b/tests/qemu-iotests/250 index af48f83aba..c0a0dbc0ff 100755 --- a/tests/qemu-iotests/250 +++ b/tests/qemu-iotests/250 @@ -52,11 +52,6 @@ _unsupported_imgopts data_file # bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which might succeed # anyway. -disk_usage() -{ - du --block-size=1 $1 | awk '{print $1}' -} - size=2100M _make_test_img -o "cluster_size=1M,preallocation=metadata" $size diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index 95c12577dd..5d2ea26c7f 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -140,6 +140,12 @@ _optstr_add() fi } +# report real disk usage for sparse files +disk_usage() +{ + du --block-size=1 $1 | awk '{print $1}' +} + # Set the variables to the empty string to turn Valgrind off # for specific processes, e.g. # $ VALGRIND_QEMU_IO= ./check -qcow2 -valgrind 015 From patchwork Fri Oct 20 21:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Drobyshev X-Patchwork-Id: 1852847 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=virtuozzo.com header.i=@virtuozzo.com header.a=rsa-sha256 header.s=relay header.b=CLcr3tGD; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SBz475fNSz23jq for ; Sat, 21 Oct 2023 08:58:31 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qtxTx-0003r7-87; Fri, 20 Oct 2023 17:56:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTu-0003pN-AB; Fri, 20 Oct 2023 17:56:35 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qtxTq-0008G2-OS; Fri, 20 Oct 2023 17:56:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-Id:Date:Subject:From: Content-Type; bh=xoAP6FRspQjKbgVizSw7Z47QzqhgL7NJIPq3ksUKjlQ=; b=CLcr3tGDzQpd 4+NuWJAQd0mk3B2QF8YmZZhYCYO8AjI4AyqJbSAYXY/Qzfvj4gpmHsQnN1dV0XBQMmsmH2FaIkP6y QJF27wyG729GUXK6sxiERxKB/sNrytvWVDpA/ZI2OuUda73bZil9kDZ3MOi6DNFgnsluX7zWZe/Lx TTQ8p5MQh3I/mmZLPn5uZ9YyWSsGskQAKyH2xspoP9ok5EhcWagJIbE59HIGCzpwiwGH61idMoSYv 5+5xv1kvxe0iey525ZoL2vZ06/G2daZd92vFvmgmm+NjhPSgpm8bKCyd7kWQq7iGW/TXBg5232wlU m5jFPanI66HdTSQUzOxkDQ==; Received: from [130.117.225.1] (helo=dev005.ch-qa.vzint.dev) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1qtxTT-00A9pF-2n; Fri, 20 Oct 2023 23:56:20 +0200 From: Andrey Drobyshev To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, hreitz@redhat.com, kwolf@redhat.com, eblake@redhat.com, berto@igalia.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 7/7] iotests/271: check disk usage on subcluster-based discard/unmap Date: Sat, 21 Oct 2023 00:56:22 +0300 Message-Id: <20231020215622.789260-8-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> References: <20231020215622.789260-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Add _verify_du_delta() checker which is used to check that real disk usage delta meets the expectations. For now we use it for checking that subcluster-based discard/unmap operations lead to actual disk usage decrease (i.e. PUNCH_HOLE operation is performed). Also add separate test case for discarding particular subcluster within one cluster. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/271 | 25 ++++++++++++++++++++++++- tests/qemu-iotests/271.out | 2 ++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271 index c7c2cadda0..5fcb209f5f 100755 --- a/tests/qemu-iotests/271 +++ b/tests/qemu-iotests/271 @@ -81,6 +81,15 @@ _verify_l2_bitmap() fi } +# Check disk usage delta after a discard/unmap operation +# _verify_du_delta $before $after $expected_delta +_verify_du_delta() +{ + if [ $(($1 - $2)) -ne $3 ]; then + printf "ERROR: unexpected delta: $1 - $2 = $(($1 - $2)) != $3\n" + fi +} + # This should be called as _run_test c=XXX sc=XXX off=XXX len=XXX cmd=XXX # c: cluster number (0 if unset) # sc: subcluster number inside cluster @c (0 if unset) @@ -198,9 +207,12 @@ for use_backing_file in yes no; do alloc="$(seq 0 31)"; zero="" _run_test sc=0 len=64k - ### Zero and unmap half of cluster #0 (this won't unmap it) + ### Zero and unmap half of cluster #0 (this will unmap it) alloc="$(seq 16 31)"; zero="$(seq 0 15)" + before=$(disk_usage "$TEST_IMG") _run_test sc=0 len=32k cmd=unmap + after=$(disk_usage "$TEST_IMG") + _verify_du_delta $before $after 32768 ### Zero and unmap cluster #0 alloc=""; zero="$(seq 0 31)" @@ -447,7 +459,10 @@ for use_backing_file in yes no; do # Subcluster-aligned request from clusters #12 to #14 alloc="$(seq 0 15)"; zero="$(seq 16 31)" + before=$(disk_usage "$TEST_IMG") _run_test c=12 sc=16 len=128k cmd=unmap + after=$(disk_usage "$TEST_IMG") + _verify_du_delta $before $after $((128 * 1024)) alloc=""; zero="$(seq 0 31)" _verify_l2_bitmap 13 alloc="$(seq 16 31)"; zero="$(seq 0 15)" @@ -528,6 +543,14 @@ for use_backing_file in yes no; do else _make_test_img -o extended_l2=on 1M fi + # Write cluster #0 and discard its subclusters #0-#3 + $QEMU_IO -c 'write -q 0 64k' "$TEST_IMG" + before=$(disk_usage "$TEST_IMG") + $QEMU_IO -c 'discard -q 0 8k' "$TEST_IMG" + after=$(disk_usage "$TEST_IMG") + _verify_du_delta $before $after 8192 + alloc="$(seq 4 31)"; zero="$(seq 0 3)" + _verify_l2_bitmap 0 # Write clusters #0-#2 and then discard them $QEMU_IO -c 'write -q 0 128k' "$TEST_IMG" $QEMU_IO -c 'discard -q 0 128k' "$TEST_IMG" diff --git a/tests/qemu-iotests/271.out b/tests/qemu-iotests/271.out index 5be780de76..0da8d72cde 100644 --- a/tests/qemu-iotests/271.out +++ b/tests/qemu-iotests/271.out @@ -426,6 +426,7 @@ L2 entry #29: 0x0000000000000000 0000ffff00000000 ### Discarding clusters with non-zero bitmaps (backing file: yes) ### Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=raw +L2 entry #0: 0x8000000000050000 0000000ffffffff0 L2 entry #0: 0x0000000000000000 ffffffff00000000 L2 entry #1: 0x0000000000000000 ffffffff00000000 Image resized. @@ -436,6 +437,7 @@ L2 entry #1: 0x0000000000000000 ffffffff00000000 ### Discarding clusters with non-zero bitmaps (backing file: no) ### Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 +L2 entry #0: 0x8000000000050000 0000000ffffffff0 L2 entry #0: 0x0000000000000000 ffffffff00000000 L2 entry #1: 0x0000000000000000 ffffffff00000000 Image resized.