From patchwork Tue Aug 12 13:42:27 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Beno=C3=AEt_Canet?= X-Patchwork-Id: 379356 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3D0A8140087 for ; Tue, 12 Aug 2014 23:53:29 +1000 (EST) Received: from localhost ([::1]:42199 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XHCVz-0000v5-68 for incoming@patchwork.ozlabs.org; Tue, 12 Aug 2014 09:53:27 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41550) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XHCMK-0000hK-Fr for qemu-devel@nongnu.org; Tue, 12 Aug 2014 09:43:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XHCMD-0001WY-9k for qemu-devel@nongnu.org; Tue, 12 Aug 2014 09:43:28 -0400 Received: from lputeaux-656-01-25-125.w80-12.abo.wanadoo.fr ([80.12.84.125]:41998 helo=paradis.irqsave.net) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XHCMC-0001Vz-KB for qemu-devel@nongnu.org; Tue, 12 Aug 2014 09:43:21 -0400 Received: from localhost.localdomain (laure.irqsave.net [192.168.77.2]) by paradis.irqsave.net (Postfix) with ESMTP id D1133CB504; Tue, 12 Aug 2014 15:43:17 +0200 (CEST) From: =?UTF-8?q?Beno=C3=AEt=20Canet?= To: qemu-devel@nongnu.org Date: Tue, 12 Aug 2014 15:42:27 +0200 Message-Id: <1407850947-17625-6-git-send-email-benoit.canet@irqsave.net> X-Mailer: git-send-email 2.1.0.rc1 In-Reply-To: <1407850947-17625-1-git-send-email-benoit.canet@irqsave.net> References: <1407850947-17625-1-git-send-email-benoit.canet@irqsave.net> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 80.12.84.125 Cc: kwolf@redhat.com, =?UTF-8?q?Beno=C3=AEt=20Canet?= , Benoit Canet , stefanha@redhat.com Subject: [Qemu-devel] [RFC 5/5] throttle: Add throttle group support. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Signed-off-by: Benoit Canet --- block.c | 157 ++++++++++++++++++++++++++++++++++++++++------ block/qapi.c | 2 +- blockdev.c | 16 ++++- hmp.c | 4 +- include/block/block.h | 2 +- include/block/block_int.h | 9 +-- include/qemu/throttle.h | 5 +- qapi/block-core.json | 5 +- qemu-options.hx | 1 + qmp-commands.hx | 3 +- util/throttle-groups.c | 2 +- util/throttle.c | 23 +++++-- 12 files changed, 192 insertions(+), 37 deletions(-) diff --git a/block.c b/block.c index 5edb3c3..92f9171 100644 --- a/block.c +++ b/block.c @@ -35,6 +35,7 @@ #include "qmp-commands.h" #include "qemu/timer.h" #include "qapi-event.h" +#include "qemu/throttle-groups.h" #ifdef CONFIG_BSD #include @@ -126,7 +127,9 @@ void bdrv_set_io_limits(BlockDriverState *bs, { int i; - throttle_config(&bs->throttle_state, &bs->throttle_timers, cfg); + throttle_group_lock(bs->throttle_state); + throttle_config(bs->throttle_state, &bs->throttle_timers, cfg); + throttle_group_unlock(bs->throttle_state); for (i = 0; i < 2; i++) { qemu_co_enter_next(&bs->throttled_reqs[i]); @@ -153,34 +156,93 @@ static bool bdrv_start_throttled_reqs(BlockDriverState *bs) return drained; } +static void bdrv_throttle_group_add(BlockDriverState *bs) +{ + int i; + BlockDriverState *token; + + for (i = 0; i < 2; i++) { + /* Get the BlockDriverState having the round robin token */ + token = throttle_group_token(bs->throttle_state, i); + + /* If the ThrottleGroup is new set the current BlockDriverState as + * token + */ + if (!token) { + throttle_group_set_token(bs->throttle_state, bs, i); + } + + } + + throttle_group_register_bs(bs->throttle_state, bs); +} + +static void bdrv_throttle_group_remove(BlockDriverState *bs) +{ + BlockDriverState *token; + int i; + + for (i = 0; i < 2; i++) { + /* Get the BlockDriverState having the round robin token */ + token = throttle_group_token(bs->throttle_state, i); + /* if this bs is the current token set the next bs as token */ + if (token == bs) { + token = throttle_group_next_bs(token); + throttle_group_set_token(bs->throttle_state, token, i); + } + } + + /* remove the current bs from the list */ + QLIST_REMOVE(bs, round_robin); +} + void bdrv_io_limits_disable(BlockDriverState *bs) { - bs->io_limits_enabled = false; + throttle_group_lock(bs->throttle_state); + + bs->io_limits_enabled = false; bdrv_start_throttled_reqs(bs); + bdrv_throttle_group_remove(bs); + + /* unlock all throttle group related elements */ + throttle_group_unlock(bs->throttle_state); + + throttle_group_unref(bs->throttle_state); + bs->throttle_state = NULL; + throttle_timers_destroy(&bs->throttle_timers); } static void bdrv_throttle_read_timer_cb(void *opaque) { BlockDriverState *bs = opaque; - throttle_timer_fired(&bs->throttle_state, false); + throttle_group_lock(bs->throttle_state); + throttle_timer_fired(bs->throttle_state, false); + throttle_group_unlock(bs->throttle_state); qemu_co_enter_next(&bs->throttled_reqs[0]); } static void bdrv_throttle_write_timer_cb(void *opaque) { BlockDriverState *bs = opaque; - throttle_timer_fired(&bs->throttle_state, true); + throttle_group_lock(bs->throttle_state); + throttle_timer_fired(bs->throttle_state, true); + throttle_group_unlock(bs->throttle_state); qemu_co_enter_next(&bs->throttled_reqs[1]); } /* should be called before bdrv_set_io_limits if a limit is set */ -void bdrv_io_limits_enable(BlockDriverState *bs) +void bdrv_io_limits_enable(BlockDriverState *bs, const char *group) { assert(!bs->io_limits_enabled); - throttle_init(&bs->throttle_state); + bs->throttle_state = throttle_group_incref(group ? group: bs->device_name); + + throttle_group_lock(bs->throttle_state); + + bdrv_throttle_group_add(bs); + throttle_timers_init(&bs->throttle_timers, bdrv_get_aio_context(bs), QEMU_CLOCK_VIRTUAL, @@ -188,6 +250,33 @@ void bdrv_io_limits_enable(BlockDriverState *bs) bdrv_throttle_write_timer_cb, bs); bs->io_limits_enabled = true; + + /* unlock all throttle group related elements */ + throttle_group_unlock(bs->throttle_state); +} + +/* This implement the round robin policy and must be called under ThrottleGroup + * lock + */ +static BlockDriverState *bdrv_next_throttle_token(BlockDriverState *bs, + bool is_write) +{ + BlockDriverState *token, *start; + + start = token = throttle_group_token(bs->throttle_state, is_write); + + token = throttle_group_next_bs(token); + while (token != start && + qemu_co_queue_empty(&token->throttled_reqs[is_write])) { + token = throttle_group_next_bs(token); + } + + if (token == start && + qemu_co_queue_empty(&token->throttled_reqs[is_write])) { + token = bs; + } + + return token; } /* This function makes an IO wait if needed @@ -199,29 +288,63 @@ static void bdrv_io_limits_intercept(BlockDriverState *bs, unsigned int bytes, bool is_write) { + bool empty; + bool armed; + bool token_queue_empty; + BlockDriverState *token; + + throttle_group_lock(bs->throttle_state); + token = bdrv_next_throttle_token(bs, is_write); + /* does this io must wait */ - bool must_wait = throttle_schedule_timer(&bs->throttle_state, - &bs->throttle_timers, - is_write); + bool must_wait = throttle_schedule_timer(bs->throttle_state, + &token->throttle_timers, + is_write, + &armed); + + empty = qemu_co_queue_empty(&bs->throttled_reqs[is_write]); + + if (armed) { + throttle_group_set_token(bs->throttle_state, token, is_write); + } + + throttle_group_unlock(bs->throttle_state); /* if must wait or any request of this type throttled queue the IO */ - if (must_wait || - !qemu_co_queue_empty(&bs->throttled_reqs[is_write])) { + if (must_wait || !empty) { qemu_co_queue_wait(&bs->throttled_reqs[is_write]); } + throttle_group_lock(bs->throttle_state); + token = bdrv_next_throttle_token(bs, is_write); + token_queue_empty = qemu_co_queue_empty(&token->throttled_reqs[is_write]); + + /* the IO will be executed, do the accounting */ - throttle_account(&bs->throttle_state, is_write, bytes); + throttle_account(bs->throttle_state, is_write, bytes); + must_wait = throttle_schedule_timer(bs->throttle_state, + &token->throttle_timers, + is_write, + &armed); + + if (armed || !token_queue_empty) { + throttle_group_set_token(bs->throttle_state, token, is_write); + } /* if the next request must wait -> do nothing */ - if (throttle_schedule_timer(&bs->throttle_state, &bs->throttle_timers, - is_write)) { + if (must_wait) { + throttle_group_unlock(bs->throttle_state); return; } /* else queue next request for execution */ - qemu_co_queue_next(&bs->throttled_reqs[is_write]); + if (!qemu_co_queue_empty(&bs->throttled_reqs[is_write])) { + qemu_co_queue_next(&bs->throttled_reqs[is_write]); + } else if(!token_queue_empty) { + throttle_arm_timer(&token->throttle_timers, is_write); + } + throttle_group_unlock(bs->throttle_state); } size_t bdrv_opt_mem_align(BlockDriverState *bs) @@ -1972,9 +2095,7 @@ static void bdrv_move_feature_fields(BlockDriverState *bs_dest, bs_dest->enable_write_cache = bs_src->enable_write_cache; /* i/o throttled req */ - memcpy(&bs_dest->throttle_state, - &bs_src->throttle_state, - sizeof(ThrottleState)); + bs_dest->throttle_state = bs_src->throttle_state, bs_dest->throttled_reqs[0] = bs_src->throttled_reqs[0]; bs_dest->throttled_reqs[1] = bs_src->throttled_reqs[1]; bs_dest->io_limits_enabled = bs_src->io_limits_enabled; diff --git a/block/qapi.c b/block/qapi.c index f44f6b4..9ea11a0 100644 --- a/block/qapi.c +++ b/block/qapi.c @@ -54,7 +54,7 @@ BlockDeviceInfo *bdrv_block_device_info(BlockDriverState *bs) if (bs->io_limits_enabled) { ThrottleConfig cfg; - throttle_get_config(&bs->throttle_state, &cfg); + throttle_get_config(bs->throttle_state, &cfg); info->bps = cfg.buckets[THROTTLE_BPS_TOTAL].avg; info->bps_rd = cfg.buckets[THROTTLE_BPS_READ].avg; info->bps_wr = cfg.buckets[THROTTLE_BPS_WRITE].avg; diff --git a/blockdev.c b/blockdev.c index 48bd9a3..67e2d29 100644 --- a/blockdev.c +++ b/blockdev.c @@ -330,6 +330,7 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts, bool has_driver_specific_opts; BlockdevDetectZeroesOptions detect_zeroes; BlockDriver *drv = NULL; + const char *throttling_group; /* Check common options by copying from bs_opts to opts, all other options * stay in bs_opts for processing by bdrv_open(). */ @@ -432,6 +433,8 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts, cfg.op_size = qemu_opt_get_number(opts, "throttling.iops-size", 0); + throttling_group = qemu_opt_get(opts, "throttling.group"); + if (!check_throttle_config(&cfg, &error)) { error_propagate(errp, error); goto early_err; @@ -490,7 +493,7 @@ static DriveInfo *blockdev_init(const char *file, QDict *bs_opts, /* disk I/O throttling */ if (throttle_enabled(&cfg)) { - bdrv_io_limits_enable(dinfo->bdrv); + bdrv_io_limits_enable(dinfo->bdrv, throttling_group); bdrv_set_io_limits(dinfo->bdrv, &cfg); } @@ -679,6 +682,7 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType block_default_type) qemu_opt_rename(all_opts, "iops_size", "throttling.iops-size"); + qemu_opt_rename(all_opts, "group", "throttling.group"); qemu_opt_rename(all_opts, "readonly", "read-only"); @@ -1689,7 +1693,9 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd, bool has_iops_wr_max, int64_t iops_wr_max, bool has_iops_size, - int64_t iops_size, Error **errp) + int64_t iops_size, + bool has_group, + const char *group, Error **errp) { ThrottleConfig cfg; BlockDriverState *bs; @@ -1741,7 +1747,7 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd, aio_context_acquire(aio_context); if (!bs->io_limits_enabled && throttle_enabled(&cfg)) { - bdrv_io_limits_enable(bs); + bdrv_io_limits_enable(bs, has_group ? group : NULL); } else if (bs->io_limits_enabled && !throttle_enabled(&cfg)) { bdrv_io_limits_disable(bs); } @@ -2643,6 +2649,10 @@ QemuOptsList qemu_common_drive_opts = { .type = QEMU_OPT_NUMBER, .help = "when limiting by iops max size of an I/O in bytes", },{ + .name = "throttling.group", + .type = QEMU_OPT_STRING, + .help = "name of the block throttling group", + },{ .name = "copy-on-read", .type = QEMU_OPT_BOOL, .help = "copy read data from backing file into image file", diff --git a/hmp.c b/hmp.c index 4d1838e..c580b0e 100644 --- a/hmp.c +++ b/hmp.c @@ -1165,7 +1165,9 @@ void hmp_block_set_io_throttle(Monitor *mon, const QDict *qdict) false, 0, false, /* No default I/O size */ - 0, &err); + 0, + false, + NULL, &err); hmp_handle_error(mon, &err); } diff --git a/include/block/block.h b/include/block/block.h index f08471d..761006d 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -190,7 +190,7 @@ void bdrv_stats_print(Monitor *mon, const QObject *data); void bdrv_info_stats(Monitor *mon, QObject **ret_data); /* disk I/O throttling */ -void bdrv_io_limits_enable(BlockDriverState *bs); +void bdrv_io_limits_enable(BlockDriverState *bs, const char *group); void bdrv_io_limits_disable(BlockDriverState *bs); void bdrv_init(void); diff --git a/include/block/block_int.h b/include/block/block_int.h index 6066f63..fbf5d2e 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -334,12 +334,13 @@ struct BlockDriverState { /* number of in-flight serialising requests */ unsigned int serialising_in_flight; - /* I/O throttling */ - ThrottleState throttle_state; - ThrottleTimers throttle_timers; - CoQueue throttled_reqs[2]; + /* I/O throttling - following elements protected by ThrottleGroup lock */ + ThrottleState *throttle_state; bool io_limits_enabled; + CoQueue throttled_reqs[2]; QLIST_ENTRY(BlockDriverState) round_robin; + /* timers have their own locking */ + ThrottleTimers throttle_timers; /* I/O stats (display with "info blockstats"). */ uint64_t nr_bytes[BDRV_MAX_IOTYPE]; diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index 3aece3a..5f5dde2 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -124,7 +124,10 @@ void throttle_get_config(ThrottleState *ts, ThrottleConfig *cfg); /* usage */ bool throttle_schedule_timer(ThrottleState *ts, ThrottleTimers *tt, - bool is_write); + bool is_write, + bool *armed); + +void throttle_arm_timer(ThrottleTimers *tt, bool is_write); void throttle_timer_fired(ThrottleState *ts, bool is_write); diff --git a/qapi/block-core.json b/qapi/block-core.json index e378653..aa307a2 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -886,6 +886,9 @@ # # @iops_size: #optional an I/O size in bytes (Since 1.7) # +# +# @group: #optional throttle group name (Since 2.2) +# # Returns: Nothing on success # If @device is not a valid block device, DeviceNotFound # @@ -897,7 +900,7 @@ '*bps_max': 'int', '*bps_rd_max': 'int', '*bps_wr_max': 'int', '*iops_max': 'int', '*iops_rd_max': 'int', '*iops_wr_max': 'int', - '*iops_size': 'int' } } + '*iops_size': 'int', '*group': 'str' } } ## # @block-stream: diff --git a/qemu-options.hx b/qemu-options.hx index 1549625..f1ca6aa 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -433,6 +433,7 @@ DEF("drive", HAS_ARG, QEMU_OPTION_drive, " [[,bps_max=bm]|[[,bps_rd_max=rm][,bps_wr_max=wm]]]\n" " [[,iops_max=im]|[[,iops_rd_max=irm][,iops_wr_max=iwm]]]\n" " [[,iops_size=is]]\n" + " [[,group=g]]\n" " use 'file' as a drive image\n", QEMU_ARCH_ALL) STEXI @item -drive @var{option}[,@var{option}[,@var{option}[,...]]] diff --git a/qmp-commands.hx b/qmp-commands.hx index 4be4765..2f25ac7 100644 --- a/qmp-commands.hx +++ b/qmp-commands.hx @@ -1663,7 +1663,7 @@ EQMP { .name = "block_set_io_throttle", - .args_type = "device:B,bps:l,bps_rd:l,bps_wr:l,iops:l,iops_rd:l,iops_wr:l,bps_max:l?,bps_rd_max:l?,bps_wr_max:l?,iops_max:l?,iops_rd_max:l?,iops_wr_max:l?,iops_size:l?", + .args_type = "device:B,bps:l,bps_rd:l,bps_wr:l,iops:l,iops_rd:l,iops_wr:l,bps_max:l?,bps_rd_max:l?,bps_wr_max:l?,iops_max:l?,iops_rd_max:l?,iops_wr_max:l?,iops_size:l?,group:s?", .mhandler.cmd_new = qmp_marshal_input_block_set_io_throttle, }, @@ -1689,6 +1689,7 @@ Arguments: - "iops_rd_max": read I/O operations max (json-int) - "iops_wr_max": write I/O operations max (json-int) - "iops_size": I/O size in bytes when limiting (json-int) +- "group": throttle group name (json-string) Example: diff --git a/util/throttle-groups.c b/util/throttle-groups.c index 153bacc..20baca4 100644 --- a/util/throttle-groups.c +++ b/util/throttle-groups.c @@ -137,7 +137,7 @@ void throttle_group_register_bs(ThrottleState *ts, BlockDriverState *bs) */ BlockDriverState *throttle_group_next_bs(BlockDriverState *bs) { - ThrottleState *ts = &bs->throttle_state; + ThrottleState *ts = bs->throttle_state; ThrottleGroup *tg = container_of(ts, ThrottleGroup, ts); BlockDriverState *next = QLIST_NEXT(bs, round_robin); diff --git a/util/throttle.c b/util/throttle.c index 0e305e3..1ee52eb 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -375,11 +375,13 @@ void throttle_get_config(ThrottleState *ts, ThrottleConfig *cfg) */ bool throttle_schedule_timer(ThrottleState *ts, ThrottleTimers *tt, - bool is_write) + bool is_write, + bool *armed) { int64_t now = qemu_clock_get_ns(tt->clock_type); int64_t next_timestamp; bool must_wait; + *armed = false; must_wait = throttle_compute_timer(ts, is_write, @@ -392,15 +394,26 @@ bool throttle_schedule_timer(ThrottleState *ts, } /* request throttled and any timer pending -> do nothing */ - if (ts->any_timer_armed[is_write]) { - return true; + if (!ts->any_timer_armed[is_write]) { + *armed = true; + ts->any_timer_armed[is_write] = true; + timer_mod(tt->timers[is_write], next_timestamp); } - ts->any_timer_armed[is_write] = true; - timer_mod(tt->timers[is_write], next_timestamp); return true; } +/* Schedule a throttle timer like a BH + * + * @tt: The timers structure + * @is_write: the type of operation (read/write) + */ +void throttle_arm_timer(ThrottleTimers *tt, bool is_write) +{ + int64_t now = qemu_clock_get_ns(tt->clock_type); + timer_mod(tt->timers[is_write], now + 1); +} + /* Remember that now timers are currently armed * * @ts: the throttle state we are working on