From patchwork Mon Sep 30 08:43:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1990813 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=aJrhtueK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHF5G1jl3z1xtM for ; Mon, 30 Sep 2024 18:45:22 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svC0g-0006uP-Bc; Mon, 30 Sep 2024 04:44:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0d-0006hw-BQ; Mon, 30 Sep 2024 04:43:59 -0400 Received: from forwardcorp1a.mail.yandex.net ([2a02:6b8:c0e:500:1:45:d181:df01]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0a-0000u8-5S; Mon, 30 Sep 2024 04:43:58 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:8a3:0:640:33b5:0]) by forwardcorp1a.mail.yandex.net (Yandex) with ESMTPS id 63F1060C23; Mon, 30 Sep 2024 11:43:49 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b737::1:2c]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id WhU2Sb1IXiE0-cyZRRwx5; Mon, 30 Sep 2024 11:43:48 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1727685828; bh=CA0ozR2MoFs8iqJMkGwT6szoTB2uzbt1omtnLOIN2UU=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=aJrhtueKuTAtkyxJ2JKTMDozG9n52O516UkIclLfZ3lUUhA4VBt9teCcpdvyUUp6B f+XNDDHNsM862fNPl1vPL42GfqY/5WjdLJCD+a+UyTrlwx9Qrm6c1N7aj1C2a61qvf h9CuHURWHjW7Oe43caiDJCWJ546mwUPY7E1eJRiw= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, peter.maydell@linaro.org, Fiona Ebner , Markus Armbruster Subject: [PULL 1/5] copy-before-write: allow specifying minimum cluster size Date: Mon, 30 Sep 2024 11:43:21 +0300 Message-Id: <20240930084325.187606-2-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240930084325.187606-1-vsementsov@yandex-team.ru> References: <20240930084325.187606-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 X-Yandex-Filter: 1 Received-SPF: pass client-ip=2a02:6b8:c0e:500:1:45:d181:df01; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1a.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fiona Ebner In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because cbw_co_pdiscard_snapshot() will align down the discard requests and thus effectively ignore then. To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source. The type 'size' (corresponding to uint64_t in C) is used in QAPI to rule out negative inputs and for consistency with already existing @cluster-size parameters. Since block_copy_calculate_cluster_size() uses int64_t for its result, a check that the input is not too large is added in block_copy_state_new() before calling it. The calculation in block_copy_calculate_cluster_size() is done in the target int64_t type. Suggested-by: Vladimir Sementsov-Ogievskiy Acked-by: Markus Armbruster (QAPI schema) Signed-off-by: Fiona Ebner Message-Id: <20240711120915.310243-2-f.ebner@proxmox.com> [vsementsov: switch version to 9.2 in QAPI doc] Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/block-copy.c | 36 ++++++++++++++++++++++++++---------- block/copy-before-write.c | 5 ++++- include/block/block-copy.h | 1 + qapi/block-core.json | 8 +++++++- 4 files changed, 38 insertions(+), 12 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index cc618e4561..93eb1b2664 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -310,6 +310,7 @@ void block_copy_set_copy_opts(BlockCopyState *s, bool use_copy_range, } static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, + int64_t min_cluster_size, Error **errp) { int ret; @@ -319,6 +320,9 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, GLOBAL_STATE_CODE(); GRAPH_RDLOCK_GUARD_MAINLOOP(); + min_cluster_size = MAX(min_cluster_size, + (int64_t)BLOCK_COPY_CLUSTER_SIZE_DEFAULT); + target_does_cow = bdrv_backing_chain_next(target); /* @@ -329,13 +333,13 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, ret = bdrv_get_info(target, &bdi); if (ret == -ENOTSUP && !target_does_cow) { /* Cluster size is not defined */ - warn_report("The target block device doesn't provide " - "information about the block size and it doesn't have a " - "backing file. The default block size of %u bytes is " - "used. If the actual block size of the target exceeds " - "this default, the backup may be unusable", - BLOCK_COPY_CLUSTER_SIZE_DEFAULT); - return BLOCK_COPY_CLUSTER_SIZE_DEFAULT; + warn_report("The target block device doesn't provide information about " + "the block size and it doesn't have a backing file. The " + "(default) block size of %" PRIi64 " bytes is used. If the " + "actual block size of the target exceeds this value, the " + "backup may be unusable", + min_cluster_size); + return min_cluster_size; } else if (ret < 0 && !target_does_cow) { error_setg_errno(errp, -ret, "Couldn't determine the cluster size of the target image, " @@ -345,16 +349,17 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, return ret; } else if (ret < 0 && target_does_cow) { /* Not fatal; just trudge on ahead. */ - return BLOCK_COPY_CLUSTER_SIZE_DEFAULT; + return min_cluster_size; } - return MAX(BLOCK_COPY_CLUSTER_SIZE_DEFAULT, bdi.cluster_size); + return MAX(min_cluster_size, bdi.cluster_size); } BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, bool discard_source, + uint64_t min_cluster_size, Error **errp) { ERRP_GUARD(); @@ -365,7 +370,18 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, GLOBAL_STATE_CODE(); - cluster_size = block_copy_calculate_cluster_size(target->bs, errp); + if (min_cluster_size > INT64_MAX) { + error_setg(errp, "min-cluster-size too large: %" PRIu64 " > %" PRIi64, + min_cluster_size, INT64_MAX); + return NULL; + } else if (min_cluster_size && !is_power_of_2(min_cluster_size)) { + error_setg(errp, "min-cluster-size needs to be a power of 2"); + return NULL; + } + + cluster_size = block_copy_calculate_cluster_size(target->bs, + (int64_t)min_cluster_size, + errp); if (cluster_size < 0) { return NULL; } diff --git a/block/copy-before-write.c b/block/copy-before-write.c index 853e01a1eb..a919b1f41b 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -417,6 +417,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp) qdict_extract_subqdict(options, NULL, "bitmap"); qdict_del(options, "on-cbw-error"); qdict_del(options, "cbw-timeout"); + qdict_del(options, "min-cluster-size"); out: visit_free(v); @@ -476,8 +477,10 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags, bs->file->bs->supported_zero_flags); s->discard_source = flags & BDRV_O_CBW_DISCARD_SOURCE; + s->bcs = block_copy_state_new(bs->file, s->target, bs, bitmap, - flags & BDRV_O_CBW_DISCARD_SOURCE, errp); + flags & BDRV_O_CBW_DISCARD_SOURCE, + opts->min_cluster_size, errp); if (!s->bcs) { error_prepend(errp, "Cannot create block-copy-state: "); return -EINVAL; diff --git a/include/block/block-copy.h b/include/block/block-copy.h index bdc703bacd..dd5cc82f3b 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -28,6 +28,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, bool discard_source, + uint64_t min_cluster_size, Error **errp); /* Function should be called prior any actual copy request */ diff --git a/qapi/block-core.json b/qapi/block-core.json index 9f6dd59298..6751022428 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4639,12 +4639,18 @@ # @on-cbw-error parameter will decide how this failure is handled. # Default 0. (Since 7.1) # +# @min-cluster-size: Minimum size of blocks used by copy-before-write +# operations. Has to be a power of 2. No effect if smaller than +# the maximum of the target's cluster size and 64 KiB. Default 0. +# (Since 9.2) +# # Since: 6.2 ## { 'struct': 'BlockdevOptionsCbw', 'base': 'BlockdevOptionsGenericFormat', 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap', - '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32' } } + '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32', + '*min-cluster-size': 'size' } } ## # @BlockdevOptions: From patchwork Mon Sep 30 08:43:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1990815 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=Wr1SkxNV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHF5m20VJz1xtM for ; Mon, 30 Sep 2024 18:45:46 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svC0i-000725-Bh; Mon, 30 Sep 2024 04:44:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0e-0006m9-BS; Mon, 30 Sep 2024 04:44:00 -0400 Received: from forwardcorp1d.mail.yandex.net ([178.154.239.200]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0b-0000tv-05; Mon, 30 Sep 2024 04:44:00 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:8a3:0:640:33b5:0]) by forwardcorp1d.mail.yandex.net (Yandex) with ESMTPS id 4C7AD60B09; Mon, 30 Sep 2024 11:43:50 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b737::1:2c]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id WhU2Sb1IXiE0-35T5nJzl; Mon, 30 Sep 2024 11:43:49 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1727685829; bh=A0n8HyYvSwxkRyzh1ThtysCFIAcf3fFl7IdICB5q2i0=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=Wr1SkxNVGNx8YFYFSVuzV773E0oExSUq6Ck4dC+C3OxvDFaOrtNMt0r7ChtjaT3UW 0enOK4FlY2o4AZvTTZrpVpbqsAqL0LUDnnrF2KmioXyj4zxsVWGmPdTXUJogp35/Kf q5lSqUpu/koOOdc2tD7QDa1PMtL4TOjLe+eQkYTU= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, peter.maydell@linaro.org, Fiona Ebner , Markus Armbruster Subject: [PULL 2/5] backup: add minimum cluster size to performance options Date: Mon, 30 Sep 2024 11:43:22 +0300 Message-Id: <20240930084325.187606-3-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240930084325.187606-1-vsementsov@yandex-team.ru> References: <20240930084325.187606-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 X-Yandex-Filter: 1 Received-SPF: pass client-ip=178.154.239.200; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1d.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fiona Ebner In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because cbw_co_pdiscard_snapshot() will align down the discard requests and thus effectively ignore then. To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source. Suggested-by: Vladimir Sementsov-Ogievskiy Acked-by: Markus Armbruster (QAPI schema) Signed-off-by: Fiona Ebner Message-Id: <20240711120915.310243-3-f.ebner@proxmox.com> [vsementsov: switch version to 9.2 in QAPI doc] Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/backup.c | 2 +- block/copy-before-write.c | 9 +++++++++ block/copy-before-write.h | 1 + blockdev.c | 3 +++ qapi/block-core.json | 9 +++++++-- 5 files changed, 21 insertions(+), 3 deletions(-) diff --git a/block/backup.c b/block/backup.c index 3dd2e229d2..a1292c01ec 100644 --- a/block/backup.c +++ b/block/backup.c @@ -458,7 +458,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, } cbw = bdrv_cbw_append(bs, target, filter_node_name, discard_source, - &bcs, errp); + perf->min_cluster_size, &bcs, errp); if (!cbw) { goto error; } diff --git a/block/copy-before-write.c b/block/copy-before-write.c index a919b1f41b..e835987e52 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -548,6 +548,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, bool discard_source, + uint64_t min_cluster_size, BlockCopyState **bcs, Error **errp) { @@ -567,6 +568,14 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, qdict_put_str(opts, "file", bdrv_get_node_name(source)); qdict_put_str(opts, "target", bdrv_get_node_name(target)); + if (min_cluster_size > INT64_MAX) { + error_setg(errp, "min-cluster-size too large: %" PRIu64 " > %" PRIi64, + min_cluster_size, INT64_MAX); + qobject_unref(opts); + return NULL; + } + qdict_put_int(opts, "min-cluster-size", (int64_t)min_cluster_size); + top = bdrv_insert_node(source, opts, flags, errp); if (!top) { return NULL; diff --git a/block/copy-before-write.h b/block/copy-before-write.h index 01af0cd3c4..2a5d4ba693 100644 --- a/block/copy-before-write.h +++ b/block/copy-before-write.h @@ -40,6 +40,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, bool discard_source, + uint64_t min_cluster_size, BlockCopyState **bcs, Error **errp); void bdrv_cbw_drop(BlockDriverState *bs); diff --git a/blockdev.c b/blockdev.c index 835064ed03..6740663fda 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2655,6 +2655,9 @@ static BlockJob *do_backup_common(BackupCommon *backup, if (backup->x_perf->has_max_chunk) { perf.max_chunk = backup->x_perf->max_chunk; } + if (backup->x_perf->has_min_cluster_size) { + perf.min_cluster_size = backup->x_perf->min_cluster_size; + } } if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) || diff --git a/qapi/block-core.json b/qapi/block-core.json index 6751022428..c3b0a2376b 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1551,11 +1551,16 @@ # it should not be less than job cluster size which is calculated # as maximum of target image cluster size and 64k. Default 0. # +# @min-cluster-size: Minimum size of blocks used by copy-before-write +# and background copy operations. Has to be a power of 2. No +# effect if smaller than the maximum of the target's cluster size +# and 64 KiB. Default 0. (Since 9.2) +# # Since: 6.0 ## { 'struct': 'BackupPerf', - 'data': { '*use-copy-range': 'bool', - '*max-workers': 'int', '*max-chunk': 'int64' } } + 'data': { '*use-copy-range': 'bool', '*max-workers': 'int', + '*max-chunk': 'int64', '*min-cluster-size': 'size' } } ## # @BackupCommon: From patchwork Mon Sep 30 08:43:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1990812 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=IiEj+Dsx; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHF5D1kljz1xtM for ; Mon, 30 Sep 2024 18:45:20 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svC0f-0006r8-JH; Mon, 30 Sep 2024 04:44:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0d-0006i6-CA; Mon, 30 Sep 2024 04:43:59 -0400 Received: from forwardcorp1a.mail.yandex.net ([178.154.239.72]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0a-0000u2-Qn; Mon, 30 Sep 2024 04:43:58 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:8a3:0:640:33b5:0]) by forwardcorp1a.mail.yandex.net (Yandex) with ESMTPS id 1FBE860C2B; Mon, 30 Sep 2024 11:43:51 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b737::1:2c]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id WhU2Sb1IXiE0-Js0BTplb; Mon, 30 Sep 2024 11:43:50 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1727685830; bh=xkLZ1GZTOtbVQt+osR2ziS0pLuj2G7hrkNYOXSXAPIU=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=IiEj+DsxciJprBGbM1C+akCKS7FPBBR3BcgHR5feuZ4oLx0voQgCmY6Lqefq/dE4C aAGRg/zNrmRiJ8cAWsEtROy62cct4etzfeQoso7g/pFW1lpKxaNtKqTY6jJiLyYtlr HKOjROJxQzsTyJCxXqfq4fcd5EJK+cvANSUtdStU= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, peter.maydell@linaro.org, Fiona Ebner , qemu-stable@nongnu.org Subject: [PULL 3/5] block/reqlist: allow adding overlapping requests Date: Mon, 30 Sep 2024 11:43:23 +0300 Message-Id: <20240930084325.187606-4-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240930084325.187606-1-vsementsov@yandex-team.ru> References: <20240930084325.187606-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 X-Yandex-Filter: 1 Received-SPF: pass client-ip=178.154.239.72; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1a.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Fiona Ebner Allow overlapping request by removing the assert that made it impossible. There are only two callers: 1. block_copy_task_create() It already asserts the very same condition before calling reqlist_init_req(). 2. cbw_snapshot_read_lock() There is no need to have read requests be non-overlapping in copy-before-write when used for snapshot-access. In fact, there was no protection against two callers of cbw_snapshot_read_lock() calling reqlist_init_req() with overlapping ranges and this could lead to an assertion failure [1]. In particular, with the reproducer script below [0], two cbw_co_snapshot_block_status() callers could race, with the second calling reqlist_init_req() before the first one finishes and removes its conflicting request. [0]: > #!/bin/bash -e > dd if=/dev/urandom of=/tmp/disk.raw bs=1M count=1024 > ./qemu-img create /tmp/fleecing.raw -f raw 1G > ( > ./qemu-system-x86_64 --qmp stdio \ > --blockdev raw,node-name=node0,file.driver=file,file.filename=/tmp/disk.raw \ > --blockdev raw,node-name=node1,file.driver=file,file.filename=/tmp/fleecing.raw \ > < {"execute": "qmp_capabilities"} > {"execute": "blockdev-add", "arguments": { "driver": "copy-before-write", "file": "node0", "target": "node1", "node-name": "node3" } } > {"execute": "blockdev-add", "arguments": { "driver": "snapshot-access", "file": "node3", "node-name": "snap0" } } > {"execute": "nbd-server-start", "arguments": {"addr": { "type": "unix", "data": { "path": "/tmp/nbd.socket" } } } } > {"execute": "block-export-add", "arguments": {"id": "exp0", "node-name": "snap0", "type": "nbd", "name": "exp0"}} > EOF > ) & > sleep 5 > while true; do > ./qemu-nbd -d /dev/nbd0 > ./qemu-nbd -c /dev/nbd0 nbd:unix:/tmp/nbd.socket:exportname=exp0 -f raw -r > nbdinfo --map 'nbd+unix:///exp0?socket=/tmp/nbd.socket' > done [1]: > #5 0x000071e5f0088eb2 in __GI___assert_fail (...) at ./assert/assert.c:101 > #6 0x0000615285438017 in reqlist_init_req (...) at ../block/reqlist.c:23 > #7 0x00006152853e2d98 in cbw_snapshot_read_lock (...) at ../block/copy-before-write.c:237 > #8 0x00006152853e3068 in cbw_co_snapshot_block_status (...) at ../block/copy-before-write.c:304 > #9 0x00006152853f4d22 in bdrv_co_snapshot_block_status (...) at ../block/io.c:3726 > #10 0x000061528543a63e in snapshot_access_co_block_status (...) at ../block/snapshot-access.c:48 > #11 0x00006152853f1a0a in bdrv_co_do_block_status (...) at ../block/io.c:2474 > #12 0x00006152853f2016 in bdrv_co_common_block_status_above (...) at ../block/io.c:2652 > #13 0x00006152853f22cf in bdrv_co_block_status_above (...) at ../block/io.c:2732 > #14 0x00006152853d9a86 in blk_co_block_status_above (...) at ../block/block-backend.c:1473 > #15 0x000061528538da6c in blockstatus_to_extents (...) at ../nbd/server.c:2374 > #16 0x000061528538deb1 in nbd_co_send_block_status (...) at ../nbd/server.c:2481 > #17 0x000061528538f424 in nbd_handle_request (...) at ../nbd/server.c:2978 > #18 0x000061528538f906 in nbd_trip (...) at ../nbd/server.c:3121 > #19 0x00006152855a7caf in coroutine_trampoline (...) at ../util/coroutine-ucontext.c:175 Cc: qemu-stable@nongnu.org Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Fiona Ebner Message-Id: <20240712140716.517911-1-f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 3 ++- block/reqlist.c | 2 -- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/block/copy-before-write.c b/block/copy-before-write.c index e835987e52..81afeff1c7 100644 --- a/block/copy-before-write.c +++ b/block/copy-before-write.c @@ -66,7 +66,8 @@ typedef struct BDRVCopyBeforeWriteState { /* * @frozen_read_reqs: current read requests for fleecing user in bs->file - * node. These areas must not be rewritten by guest. + * node. These areas must not be rewritten by guest. There can be multiple + * overlapping read requests. */ BlockReqList frozen_read_reqs; diff --git a/block/reqlist.c b/block/reqlist.c index 08cb57cfa4..098e807378 100644 --- a/block/reqlist.c +++ b/block/reqlist.c @@ -20,8 +20,6 @@ void reqlist_init_req(BlockReqList *reqs, BlockReq *req, int64_t offset, int64_t bytes) { - assert(!reqlist_find_conflict(reqs, offset, bytes)); - *req = (BlockReq) { .offset = offset, .bytes = bytes, From patchwork Mon Sep 30 08:43:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1990814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=TlW670tc; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHF5b2q7zz1xtM for ; Mon, 30 Sep 2024 18:45:39 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svC0i-000744-IP; Mon, 30 Sep 2024 04:44:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0e-0006km-15; Mon, 30 Sep 2024 04:44:00 -0400 Received: from forwardcorp1d.mail.yandex.net ([178.154.239.200]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0a-0000u6-Iv; Mon, 30 Sep 2024 04:43:59 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:8a3:0:640:33b5:0]) by forwardcorp1d.mail.yandex.net (Yandex) with ESMTPS id 04BBA60B1B; Mon, 30 Sep 2024 11:43:52 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b737::1:2c]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id WhU2Sb1IXiE0-yFvb1rX8; Mon, 30 Sep 2024 11:43:51 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1727685831; bh=TNo371By6R1gWLzBQzjueDMSgHamik/HswYQWJjXTkA=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=TlW670tc3oluM5H8yvUrvl2/Fv8iBy52XtuR9zrTrqOzK0ioaxav585Fvy3XA4NYN cG/gqUaXxY63CEp354351Z/IgXrKvlEmEVZ5nZ4TOPcGg3l5bLfZOcN4Lth29/h2Wk Tkg6h5qd41Hd0c0RBfCUJ/650espKjHCEJ1AbQ5w= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, peter.maydell@linaro.org, "Dr. David Alan Gilbert" Subject: [PULL 4/5] block: Remove unused aio_task_pool_empty Date: Mon, 30 Sep 2024 11:43:24 +0300 Message-Id: <20240930084325.187606-5-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240930084325.187606-1-vsementsov@yandex-team.ru> References: <20240930084325.187606-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 X-Yandex-Filter: 1 Received-SPF: pass client-ip=178.154.239.200; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1d.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Dr. David Alan Gilbert" aio_task_pool_empty has been unused since it was added in 6e9b225f73 ("block: introduce aio task pool") Remove it. Signed-off-by: Dr. David Alan Gilbert Message-Id: <20240917002007.330689-1-dave@treblig.org> Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/aio_task.c | 5 ----- include/block/aio_task.h | 2 -- 2 files changed, 7 deletions(-) diff --git a/block/aio_task.c b/block/aio_task.c index 9bd17ea2c1..bb5c05f455 100644 --- a/block/aio_task.c +++ b/block/aio_task.c @@ -119,8 +119,3 @@ int aio_task_pool_status(AioTaskPool *pool) return pool->status; } - -bool aio_task_pool_empty(AioTaskPool *pool) -{ - return pool->busy_tasks == 0; -} diff --git a/include/block/aio_task.h b/include/block/aio_task.h index 18a9c41f4e..c81d637617 100644 --- a/include/block/aio_task.h +++ b/include/block/aio_task.h @@ -40,8 +40,6 @@ void aio_task_pool_free(AioTaskPool *); /* error code of failed task or 0 if all is OK */ int aio_task_pool_status(AioTaskPool *pool); -bool aio_task_pool_empty(AioTaskPool *pool); - /* User provides filled @task, however task->pool will be set automatically */ void coroutine_fn aio_task_pool_start_task(AioTaskPool *pool, AioTask *task); From patchwork Mon Sep 30 08:43:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 1990811 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.ru header.i=@yandex-team.ru header.a=rsa-sha256 header.s=default header.b=BuPsJ54n; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHF571SWvz1xtM for ; Mon, 30 Sep 2024 18:45:15 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1svC0h-0006yT-4V; Mon, 30 Sep 2024 04:44:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0d-0006ka-Vf; Mon, 30 Sep 2024 04:43:59 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1svC0a-0000uF-Uv; Mon, 30 Sep 2024 04:43:59 -0400 Received: from mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:8a3:0:640:33b5:0]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTPS id B6FA260E17; Mon, 30 Sep 2024 11:43:52 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6b8:b081:b737::1:2c]) by mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id WhU2Sb1IXiE0-HZoBTFZ6; Mon, 30 Sep 2024 11:43:52 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1727685832; bh=BtuTV4i6Dso4pgdFfeN1B9eAAXqbLI7RCSe0Zk64D1I=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=BuPsJ54nBlfwlxyF2AWy/MH6C55VhSN5jT7NSxyJhGv8yhe2J3j8mdztHt+Uwwk1w rP7cR33zUiouI5Jlou69dYydHGLngeqMKj7kMAnRQ9Ibt94MahLkC1yt+5YQ4zf7WP lnk8Dz98jfFEgRpC1qY2JS0xxb/EsIEUf6BQjzCg= Authentication-Results: mail-nwsmtp-smtp-corp-main-66.iva.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, vsementsov@yandex-team.ru, peter.maydell@linaro.org, "Dr. David Alan Gilbert" Subject: [PULL 5/5] util/co-shared-resource: Remove unused co_try_get_from_shres Date: Mon, 30 Sep 2024 11:43:25 +0300 Message-Id: <20240930084325.187606-6-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240930084325.187606-1-vsementsov@yandex-team.ru> References: <20240930084325.187606-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 X-Yandex-Filter: 1 Received-SPF: pass client-ip=178.154.239.136; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Dr. David Alan Gilbert" co_try_get_from_shres hasn't been used since it was added in 55fa54a789 ("co-shared-resource: protect with a mutex") (Everyone uses the _locked version) Remove it. Signed-off-by: Dr. David Alan Gilbert Message-Id: <20240918124220.27871-1-dave@treblig.org> Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/co-shared-resource.h | 7 ------- util/qemu-co-shared-resource.c | 6 ------ 2 files changed, 13 deletions(-) diff --git a/include/qemu/co-shared-resource.h b/include/qemu/co-shared-resource.h index 78ca5850f8..41be1a8131 100644 --- a/include/qemu/co-shared-resource.h +++ b/include/qemu/co-shared-resource.h @@ -44,13 +44,6 @@ SharedResource *shres_create(uint64_t total); */ void shres_destroy(SharedResource *s); -/* - * Try to allocate an amount of @n. Return true on success, and false - * if there is too little left of the collective resource to fulfill - * the request. - */ -bool co_try_get_from_shres(SharedResource *s, uint64_t n); - /* * Allocate an amount of @n, and, if necessary, yield until * that becomes possible. diff --git a/util/qemu-co-shared-resource.c b/util/qemu-co-shared-resource.c index a66cc07e75..752eb5a1c5 100644 --- a/util/qemu-co-shared-resource.c +++ b/util/qemu-co-shared-resource.c @@ -66,12 +66,6 @@ static bool co_try_get_from_shres_locked(SharedResource *s, uint64_t n) return false; } -bool co_try_get_from_shres(SharedResource *s, uint64_t n) -{ - QEMU_LOCK_GUARD(&s->lock); - return co_try_get_from_shres_locked(s, n); -} - void coroutine_fn co_get_from_shres(SharedResource *s, uint64_t n) { assert(n <= s->total);