From patchwork Mon Oct 30 16:33:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1857189 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=eU++Bq7N; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SJzQc3Pv1z1yQ4 for ; Tue, 31 Oct 2023 03:35:20 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxVDV-0003e1-17; Mon, 30 Oct 2023 12:34:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDU-0003dU-6H for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDS-0002hQ-Gl for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698683654; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pVnr5wFaay66KVUCJQ7STeYMkubTSR3EFWIQEUk4fwM=; b=eU++Bq7NO9QjaAGPe/SMrF1LXvuNvT7cOYadZ7np/y0Rfass36ia9lUUiW0ocpcsgHcZI3 USufyufMDYpBFkMcrQRQBj7j2+eDqnTm3orLv8sFhlfm8cj96wM1Jtgc2heneUoiGHhY7U FzWmhbqFHbb3EyO47yAuHQ6xJwbjJLA= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-UmzJ7G_lNr6b1ZUQ_kVFPA-1; Mon, 30 Oct 2023 12:33:56 -0400 X-MC-Unique: UmzJ7G_lNr6b1ZUQ_kVFPA-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-41e1899175eso11347661cf.1 for ; Mon, 30 Oct 2023 09:33:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683636; x=1699288436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pVnr5wFaay66KVUCJQ7STeYMkubTSR3EFWIQEUk4fwM=; b=MVSrfnEWw38cCYEIgt9tuzI2yi6iGQ7fHJhlDcEDJqIB5yAZ4+76PoiRn0CKknpeel MKCcvoTAVNyLrVaeSW6CX6r0XJnxvNmKh3y32k59/G9my0VM0EjFJgJ4PXlHC+uEQFfq 85347ZgnUl5TWc7HAJqySr2aElt+pZArtALEf+3FxB2oSnLJ/ou+Gjy9G5EL+RBXYqIV jlK/V/3vrFPMthnt1sy+qHAAnMTVxzQC1JhwFuGa2nuNfLpcVxtQPovQy4WHmlARaqbu 36ZVok5XPk+a/kLMmOsivF5oAp8SuMrMi8io4w7tQ9UjKNS6TBOGhP/XUp9bAk8odbzj AHsA== X-Gm-Message-State: AOJu0YxijV7t/6wDz9hYOv3pdkKqlLq5Inh4bUGxX2s/aG1NTnePgzPY HwvTbPIWSwV1jCFLLok7bHVMUYmNWHabf3hJ74w+u1D3bycNm1CeHyBolXBqYM/25kwxT8QK6yJ KpPFPKSrNliyPvGRKAs5Knq6V25TpuDBqzFzgrkWRlqBIPN9OWkFAJPQ1g/7ydadnG2zgAdvc X-Received: by 2002:a05:622a:148c:b0:41c:d528:6589 with SMTP id t12-20020a05622a148c00b0041cd5286589mr12742045qtx.4.1698683636281; Mon, 30 Oct 2023 09:33:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkYpWAuy0odiGhiW6g70RRy57qBHQKCXECrCTuL1GOGzldNAUgeJW4Dp9DPF2Y1akytBjTng== X-Received: by 2002:a05:622a:148c:b0:41c:d528:6589 with SMTP id t12-20020a05622a148c00b0041cd5286589mr12742020qtx.4.1698683635917; Mon, 30 Oct 2023 09:33:55 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id l15-20020ac84ccf000000b004198ae7f841sm3531111qtv.90.2023.10.30.09.33.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:33:55 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas , Joao Martins , Juan Quintela Subject: [PATCH v2 1/5] migration: Set downtime_start even for postcopy Date: Mon, 30 Oct 2023 12:33:42 -0400 Message-ID: <20231030163346.765724-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231030163346.765724-1-peterx@redhat.com> References: <20231030163346.765724-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.483, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Postcopy calculates its downtime separately. It always sets MigrationState.downtime properly, but not MigrationState.downtime_start. Make postcopy do the same as other modes on properly recording the timestamp when the VM is going to be stopped. Drop the temporary variable in postcopy_start() along the way. Signed-off-by: Peter Xu Reviewed-by: Juan Quintela --- migration/migration.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 67547eb6a1..f8a54ff4d1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2121,7 +2121,6 @@ static int postcopy_start(MigrationState *ms, Error **errp) int ret; QIOChannelBuffer *bioc; QEMUFile *fb; - int64_t time_at_stop = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); uint64_t bandwidth = migrate_max_postcopy_bandwidth(); bool restart_block = false; int cur_state = MIGRATION_STATUS_ACTIVE; @@ -2143,6 +2142,8 @@ static int postcopy_start(MigrationState *ms, Error **errp) qemu_mutex_lock_iothread(); trace_postcopy_start_set_run(); + ms->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); global_state_store(); ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); @@ -2245,7 +2246,7 @@ static int postcopy_start(MigrationState *ms, Error **errp) ms->postcopy_after_devices = true; migration_call_notifiers(ms); - ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - time_at_stop; + ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - ms->downtime_start; qemu_mutex_unlock_iothread(); From patchwork Mon Oct 30 16:33:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1857185 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FXlX7gNp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SJzPm3wxjz1yQ6 for ; Tue, 31 Oct 2023 03:34:36 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxVDI-0003ZO-SH; Mon, 30 Oct 2023 12:34:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDG-0003Yp-Fx for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:02 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDE-0002eR-RP for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698683639; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zSHyWtkUKU/y1HHFJ9tbynjlp9Czisd65damm2Zn+wQ=; b=FXlX7gNpi57jL8Jeb0OcoKt9oh9kCRAgwtlJGSTMRVpopF4PQaQ75RysB94J+n/R1TMtbV u/pJXnb7vrLmiVD5ruMrrBiZkhZT664nNLmUdMnO7m8+ATr98WnaFK1i5e59vbck0MU7+2 5dH/8ZCZ5Ue25y79qp9a39u/HL0TN6E= Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-557-PNloeZQdPHyH7xTJ2mUz1g-1; Mon, 30 Oct 2023 12:33:58 -0400 X-MC-Unique: PNloeZQdPHyH7xTJ2mUz1g-1 Received: by mail-ot1-f72.google.com with SMTP id 46e09a7af769-6cd01bd39a3so732996a34.1 for ; Mon, 30 Oct 2023 09:33:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683637; x=1699288437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zSHyWtkUKU/y1HHFJ9tbynjlp9Czisd65damm2Zn+wQ=; b=FM2vgfZyvUPxmVC+cXxVekN9DuhAQ0R3MY3K7/6Jx+l41NrUglnhb7k6T+h0D6YGKJ 8Ky9QlvnmzpLgo5f2x4KSJaKxVIkLVUWEuzbWCgAXvKzWIxt5cJChyc/5AkUjr1p6JKm I3amwK2x6SPmBNUHupiweBLC3Cjscrt1ltSORWP0urKqulTWiScAPgZe6P2MwLjRBlM7 AikbK7Lnwh9Z4THPYqLLiVINP/fK8FVnHeJqH476ddCjxld420BsYdIgc6XTHavK+Zx7 2lQyArehvBpb+G5zsQIBXdsNROrRF7mEuu8jR6dFfLVyxW7kvJw1LdJvgVrzQ2pH7YII DCeQ== X-Gm-Message-State: AOJu0YzG26fj8TcL8MBPBcpYYsRD/kBkrIcoOlWegHNs2WkJ/OtIJcdK CVvr9aJZpTFTI/SemmErPvmvyx6sguZFPZMGMHu0fE5i89zsVLmh5UnOzHeh8FP4fzyZFbJK0L3 IcnLV+OdISQaflEY/wkGiAtiZptaGm1GJijnonodMC6qSvSWvsal/6bOuBdiiK9ebg6rpLX07 X-Received: by 2002:a9d:7445:0:b0:6bf:500f:b570 with SMTP id p5-20020a9d7445000000b006bf500fb570mr10633354otk.3.1698683637381; Mon, 30 Oct 2023 09:33:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEyjoANxGj/zT43GiSr7q/8oDR4PwXA08T8ixjNi+aqzjhRpxhx5FZHE5eabifWPteJfOUVzQ== X-Received: by 2002:a9d:7445:0:b0:6bf:500f:b570 with SMTP id p5-20020a9d7445000000b006bf500fb570mr10633333otk.3.1698683637003; Mon, 30 Oct 2023 09:33:57 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id l15-20020ac84ccf000000b004198ae7f841sm3531111qtv.90.2023.10.30.09.33.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:33:56 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas , Joao Martins , Juan Quintela Subject: [PATCH v2 2/5] migration: Add migration_downtime_start|end() helpers Date: Mon, 30 Oct 2023 12:33:43 -0400 Message-ID: <20231030163346.765724-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231030163346.765724-1-peterx@redhat.com> References: <20231030163346.765724-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.483, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SORBS_WEB=1.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Unify the three users on recording downtimes with the same pair of helpers. Signed-off-by: Peter Xu Reviewed-by: Juan Quintela --- migration/migration.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index f8a54ff4d1..70d775942a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -101,6 +101,24 @@ static int migration_maybe_pause(MigrationState *s, static void migrate_fd_cancel(MigrationState *s); static int close_return_path_on_source(MigrationState *s); +static void migration_downtime_start(MigrationState *s) +{ + s->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); +} + +static void migration_downtime_end(MigrationState *s) +{ + int64_t now = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + + /* + * If downtime already set, should mean that postcopy already set it, + * then that should be the real downtime already. + */ + if (!s->downtime) { + s->downtime = now - s->downtime_start; + } +} + static bool migration_needs_multiple_sockets(void) { return migrate_multifd() || migrate_postcopy_preempt(); @@ -2142,7 +2160,7 @@ static int postcopy_start(MigrationState *ms, Error **errp) qemu_mutex_lock_iothread(); trace_postcopy_start_set_run(); - ms->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + migration_downtime_start(ms); qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); global_state_store(); @@ -2246,7 +2264,7 @@ static int postcopy_start(MigrationState *ms, Error **errp) ms->postcopy_after_devices = true; migration_call_notifiers(ms); - ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - ms->downtime_start; + migration_downtime_end(ms); qemu_mutex_unlock_iothread(); @@ -2342,7 +2360,7 @@ static int migration_completion_precopy(MigrationState *s, int ret; qemu_mutex_lock_iothread(); - s->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + migration_downtime_start(s); qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); s->vm_old_state = runstate_get(); @@ -2699,15 +2717,8 @@ static void migration_calculate_complete(MigrationState *s) int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); int64_t transfer_time; + migration_downtime_end(s); s->total_time = end_time - s->start_time; - if (!s->downtime) { - /* - * It's still not set, so we are precopy migration. For - * postcopy, downtime is calculated during postcopy_start(). - */ - s->downtime = end_time - s->downtime_start; - } - transfer_time = s->total_time - s->setup_time; if (transfer_time) { s->mbps = ((double) bytes * 8.0) / transfer_time / 1000; @@ -3126,7 +3137,7 @@ static void bg_migration_vm_start_bh(void *opaque) s->vm_start_bh = NULL; vm_start(); - s->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - s->downtime_start; + migration_downtime_end(s); } /** @@ -3193,7 +3204,7 @@ static void *bg_migration_thread(void *opaque) s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; trace_migration_thread_setup_complete(); - s->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + migration_downtime_start(s); qemu_mutex_lock_iothread(); From patchwork Mon Oct 30 16:33:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1857186 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=A2dD5DdM; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SJzPt5JjHz1yQ6 for ; Tue, 31 Oct 2023 03:34:42 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxVDL-0003bu-D3; Mon, 30 Oct 2023 12:34:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDH-0003ZC-LZ for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDF-0002ei-SE for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698683640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n72f+wW5+nHH0hrvaXv+6sujX9qgSyymTozlBQoTr/0=; b=A2dD5DdMctkeJfhaWqZHADNLtkHOxF4UGbPTgx2bUlR6/ncWPSow4IPxNHILE5KPppYQO7 b0plHrnS1Q+goGqXUz1Ea9qmHw7Ty6az7uieAydGACQKheaad/0YB9HClHT4UIkRj16InT ISHSq/ZGWX6zBB3Rdu5pyQPCFLqzPPs= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-577-w5zZZP9sOweHN-sd_gcNVw-1; Mon, 30 Oct 2023 12:33:59 -0400 X-MC-Unique: w5zZZP9sOweHN-sd_gcNVw-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-41cd9e05c8bso5543201cf.1 for ; Mon, 30 Oct 2023 09:33:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683638; x=1699288438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n72f+wW5+nHH0hrvaXv+6sujX9qgSyymTozlBQoTr/0=; b=d5XtYsTzevzLHUr0LM7dazPMhR9zmMco4ay1nFdpCvMW8b0HzlTRUmzFmtilA/7Ks9 UHDNGxkAHWNb3hyLkZyz3ZBqOPdn4oGnFu+Ds2CsrfeGkDvJ9wk4MTh1xABU7LLhDrSy KizvZaVhOWmXMnNCY8gfm71ZFhfXScZh8MEV7dWYlLtY1feX8i5srVhqt72IsWbcifqR eWZpQ7bx34hNxzNAwbRB2adTM9fJwJkGLAemxSlJaCArxfXJJwe3K7ruVbmtQrKCO3km dieycKQryGbqoDwxMk0yRj9M03hQob20J3bEcDdainTlx+BqCD4K1HPJuWBbtccJJVtA Zxnw== X-Gm-Message-State: AOJu0YyEa+emOmdiZRUlqUW+oKPE91J1YuvoI6Q/C8Ldi9MP7pPZSL4b YsFFIxBbbcfWkOKb3KvvXxBFa/gmwPwZgLEAaOxPIy4WcUKabn271hPgEIox1bLfT5jk0Y1c8i+ nHZ/duBAFWKnkIzkqm9ZWQEsl7gm+a5pleeN7qMnRwn1Y0Z0F7798BDP2frp1YFOxwp6xSbdN X-Received: by 2002:ac8:7741:0:b0:41e:49e9:fb18 with SMTP id g1-20020ac87741000000b0041e49e9fb18mr9410865qtu.0.1698683638471; Mon, 30 Oct 2023 09:33:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEiYuxKmquYKgUDZeKHAEAqk4Rqrmvcd22fxDfcMkno5M9zWKArIcp/0/Vi+MF3CudfyG9mvQ== X-Received: by 2002:ac8:7741:0:b0:41e:49e9:fb18 with SMTP id g1-20020ac87741000000b0041e49e9fb18mr9410846qtu.0.1698683638093; Mon, 30 Oct 2023 09:33:58 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id l15-20020ac84ccf000000b004198ae7f841sm3531111qtv.90.2023.10.30.09.33.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:33:57 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas , Joao Martins , Juan Quintela Subject: [PATCH v2 3/5] migration: Add per vmstate downtime tracepoints Date: Mon, 30 Oct 2023 12:33:44 -0400 Message-ID: <20231030163346.765724-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231030163346.765724-1-peterx@redhat.com> References: <20231030163346.765724-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.483, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We have a bunch of savevm_section* tracepoints, they're good to analyze migration stream, but not always suitable if someone would like to analyze the migration downtime. Two major problems: - savevm_section* tracepoints are dumping all sections, we only care about the sections that contribute to the downtime - They don't have an identifier to show the type of sections, so no way to filter downtime information either easily. We can add type into the tracepoints, but instead of doing so, this patch kept them untouched, instead of adding a bunch of downtime specific tracepoints, so one can enable "vmstate_downtime*" tracepoints and get a full picture of how the downtime is distributed across iterative and non-iterative vmstate save/load. Note that here both save() and load() need to be traced, because both of them may contribute to the downtime. The contribution is not a simple "add them together", though: consider when the src is doing a save() of device1 while the dest can be load()ing for device2, so they can happen concurrently. Tracking both sides make sense because device load() and save() can be imbalanced, one device can save() super fast, but load() super slow, vice versa. We can't figure that out without tracing both. Signed-off-by: Peter Xu Reviewed-by: Juan Quintela --- migration/savevm.c | 49 ++++++++++++++++++++++++++++++++++++++---- migration/trace-events | 2 ++ 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/migration/savevm.c b/migration/savevm.c index 8622f229e5..cd6d6ba493 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1459,6 +1459,7 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f) static int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy) { + int64_t start_ts_each, end_ts_each; SaveStateEntry *se; int ret; @@ -1475,6 +1476,8 @@ int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy) continue; } } + + start_ts_each = qemu_clock_get_us(QEMU_CLOCK_REALTIME); trace_savevm_section_start(se->idstr, se->section_id); save_section_header(f, se, QEMU_VM_SECTION_END); @@ -1486,6 +1489,9 @@ int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy) qemu_file_set_error(f, ret); return -1; } + end_ts_each = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + trace_vmstate_downtime_save("iterable", se->idstr, se->instance_id, + end_ts_each - start_ts_each); } return 0; @@ -1496,6 +1502,7 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, bool inactivate_disks) { MigrationState *ms = migrate_get_current(); + int64_t start_ts_each, end_ts_each; JSONWriter *vmdesc = ms->vmdesc; int vmdesc_len; SaveStateEntry *se; @@ -1507,11 +1514,17 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, continue; } + start_ts_each = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + ret = vmstate_save(f, se, vmdesc); if (ret) { qemu_file_set_error(f, ret); return ret; } + + end_ts_each = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + trace_vmstate_downtime_save("non-iterable", se->idstr, se->instance_id, + end_ts_each - start_ts_each); } if (inactivate_disks) { @@ -2506,9 +2519,12 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se) } static int -qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis) +qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis, + uint8_t type) { + bool trace_downtime = (type == QEMU_VM_SECTION_FULL); uint32_t instance_id, version_id, section_id; + int64_t start_ts, end_ts; SaveStateEntry *se; char idstr[256]; int ret; @@ -2557,12 +2573,23 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis) return -EINVAL; } + if (trace_downtime) { + start_ts = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + } + ret = vmstate_load(f, se); if (ret < 0) { error_report("error while loading state for instance 0x%"PRIx32" of" " device '%s'", instance_id, idstr); return ret; } + + if (trace_downtime) { + end_ts = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + trace_vmstate_downtime_load("non-iterable", se->idstr, + se->instance_id, end_ts - start_ts); + } + if (!check_section_footer(f, se)) { return -EINVAL; } @@ -2571,8 +2598,11 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis) } static int -qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis) +qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis, + uint8_t type) { + bool trace_downtime = (type == QEMU_VM_SECTION_END); + int64_t start_ts, end_ts; uint32_t section_id; SaveStateEntry *se; int ret; @@ -2597,12 +2627,23 @@ qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis) return -EINVAL; } + if (trace_downtime) { + start_ts = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + } + ret = vmstate_load(f, se); if (ret < 0) { error_report("error while loading state section id %d(%s)", section_id, se->idstr); return ret; } + + if (trace_downtime) { + end_ts = qemu_clock_get_us(QEMU_CLOCK_REALTIME); + trace_vmstate_downtime_load("iterable", se->idstr, + se->instance_id, end_ts - start_ts); + } + if (!check_section_footer(f, se)) { return -EINVAL; } @@ -2791,14 +2832,14 @@ retry: switch (section_type) { case QEMU_VM_SECTION_START: case QEMU_VM_SECTION_FULL: - ret = qemu_loadvm_section_start_full(f, mis); + ret = qemu_loadvm_section_start_full(f, mis, section_type); if (ret < 0) { goto out; } break; case QEMU_VM_SECTION_PART: case QEMU_VM_SECTION_END: - ret = qemu_loadvm_section_part_end(f, mis); + ret = qemu_loadvm_section_part_end(f, mis, section_type); if (ret < 0) { goto out; } diff --git a/migration/trace-events b/migration/trace-events index fa9486dffe..5820add1f3 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -48,6 +48,8 @@ savevm_state_cleanup(void) "" savevm_state_complete_precopy(void) "" vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s" +vmstate_downtime_save(const char *type, const char *idstr, uint32_t instance_id, int64_t downtime) "type=%s idstr=%s instance_id=%d downtime=%"PRIi64 +vmstate_downtime_load(const char *type, const char *idstr, uint32_t instance_id, int64_t downtime) "type=%s idstr=%s instance_id=%d downtime=%"PRIi64 postcopy_pause_incoming(void) "" postcopy_pause_incoming_continued(void) "" postcopy_page_req_sync(void *host_addr) "sync page req %p" From patchwork Mon Oct 30 16:33:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1857188 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=VkjYMt+B; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SJzQ84qcxz1yQ6 for ; Tue, 31 Oct 2023 03:34:56 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxVDV-0003e2-7J; Mon, 30 Oct 2023 12:34:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDT-0003dJ-07 for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDR-0002hM-IL for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698683652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7/sRV6J8eLHpB7qWOrvwuQ/IwzVXUDAHsjZN95QdXXQ=; b=VkjYMt+B4d5aqIZL9caQ0ZZjBj+UcK4hQMd5YQhphm9IvQ2vA4FEiZMV5fQ2zsbgSEBR4S 9mhc+apsIiSTa2ESDYTZQ/2UBotJdfTh/EiU0hj6xhlLWS6zcrTz//xKJmVN2u3mANS+sl hGKX2SsF6aMBOfUn3Uk0Xt6o/6+p6Kg= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-301-N7fKDIjAP_SaRLSe_okv3w-1; Mon, 30 Oct 2023 12:34:00 -0400 X-MC-Unique: N7fKDIjAP_SaRLSe_okv3w-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-4197468d5caso11355231cf.1 for ; Mon, 30 Oct 2023 09:34:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683639; x=1699288439; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7/sRV6J8eLHpB7qWOrvwuQ/IwzVXUDAHsjZN95QdXXQ=; b=w6ENlEaDEzXAQfhg4Za/FPF6ARwGn3GperMjevOp1rXNg3u7HGzPGG2aN0XOq7W8I1 n3gnEM3GkH7N+Gqb81QtNvPw2KX3Sxr9a2IOkJ9oLiF7XANaSsqhJhhcOc8B/NUjKIKY NmwjQh8uCwhyMVPg7Z1of+nwxPQ5l4h+j9c9zs4An+xgnJIdbO3w1BncJZ6khsIP74qi oy0SiRMqrmFg3v0Ik0XhUzeunK57+sIv5ETqGJz1jCNt4pkKxlLO6mwSVCw+43FsPsf0 lwFNYPpHgcROnQ06FVWBlsbQVqkh5KHWSo6eJ4aSbqkbN4bj4g0jczNGlaVZRZAzoBNU TG8g== X-Gm-Message-State: AOJu0YzyiiWRT/d9g3qbyG2yqMPHLr32pk2JatgJFRTRjm++5IM8WSNp 1McTHzhGGtzAeHLQ7zTxp4897HpXboNknLXWYL5Y0b+74pduv8WzLK+VTG0gPxxmdQw5gFS9DG8 NQ2l+Fc9NxSAHu0vAziwrlNAtbU4BaiZ6ZYkNz+LctxfP4nUqfMeP/NTQ83MCs2WqPIcCzroG X-Received: by 2002:ac8:5f11:0:b0:41c:d433:6c86 with SMTP id x17-20020ac85f11000000b0041cd4336c86mr11933745qta.4.1698683639143; Mon, 30 Oct 2023 09:33:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEUOfS9rdf4/DyVkTUOT7PJlZ8xvhIff0whlVDZ6LZHPUqdMQe47zmWSqnYWpVsJA73o6Dvjg== X-Received: by 2002:ac8:5f11:0:b0:41c:d433:6c86 with SMTP id x17-20020ac85f11000000b0041cd4336c86mr11933721qta.4.1698683638846; Mon, 30 Oct 2023 09:33:58 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id l15-20020ac84ccf000000b004198ae7f841sm3531111qtv.90.2023.10.30.09.33.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:33:58 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas , Joao Martins , Juan Quintela Subject: [PATCH v2 4/5] migration: migration_stop_vm() helper Date: Mon, 30 Oct 2023 12:33:45 -0400 Message-ID: <20231030163346.765724-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231030163346.765724-1-peterx@redhat.com> References: <20231030163346.765724-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.483, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Provide a helper for non-COLO use case of migration to stop a VM. This prepares for adding some downtime relevant tracepoints to migration, where they may or may not apply to COLO. Signed-off-by: Peter Xu Reviewed-by: Juan Quintela --- migration/migration.h | 2 ++ migration/migration.c | 11 ++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index ae82004892..5944107ad5 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -544,4 +544,6 @@ void migration_rp_wait(MigrationState *s); */ void migration_rp_kick(MigrationState *s); +int migration_stop_vm(RunState state); + #endif diff --git a/migration/migration.c b/migration/migration.c index 70d775942a..9013c1b500 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -149,6 +149,11 @@ static gint page_request_addr_cmp(gconstpointer ap, gconstpointer bp) return (a > b) - (a < b); } +int migration_stop_vm(RunState state) +{ + return vm_stop_force_state(state); +} + void migration_object_init(void) { /* This can only be called once. */ @@ -2164,7 +2169,7 @@ static int postcopy_start(MigrationState *ms, Error **errp) qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER, NULL); global_state_store(); - ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); + ret = migration_stop_vm(RUN_STATE_FINISH_MIGRATE); if (ret < 0) { goto fail; } @@ -2366,7 +2371,7 @@ static int migration_completion_precopy(MigrationState *s, s->vm_old_state = runstate_get(); global_state_store(); - ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); + ret = migration_stop_vm(RUN_STATE_FINISH_MIGRATE); trace_migration_completion_vm_stop(ret); if (ret < 0) { goto out_unlock; @@ -3217,7 +3222,7 @@ static void *bg_migration_thread(void *opaque) global_state_store(); /* Forcibly stop VM before saving state of vCPUs and devices */ - if (vm_stop_force_state(RUN_STATE_PAUSED)) { + if (migration_stop_vm(RUN_STATE_PAUSED)) { goto fail; } /* From patchwork Mon Oct 30 16:33:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1857187 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Jqre0Nyu; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SJzQ44kGSz1yQ6 for ; Tue, 31 Oct 2023 03:34:52 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qxVDK-0003Zp-I5; Mon, 30 Oct 2023 12:34:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDJ-0003Zd-2H for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qxVDH-0002f7-6L for qemu-devel@nongnu.org; Mon, 30 Oct 2023 12:34:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698683642; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H7LWaJqNMU4MMH18+1IMpyA0GlKWGvM59R3vPm4Tplc=; b=Jqre0NyuEp63j0e/Ji1Q38yZlgRvw+sJ5HTl487WvrTsn+OyfjzPnp9GLBS10aH9LKsByk kPK+Ecu4PqcpteYfvnc57sRcHe5PSxtmTWS/RsJPZ9yDe+iK9Js4aplwowNFSK0BHaz/nc fB934LKm8xGukSYTa2NOeA/EpHzajqc= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-661-aEzSSJuNNbeeSMFUs69s4g-1; Mon, 30 Oct 2023 12:34:01 -0400 X-MC-Unique: aEzSSJuNNbeeSMFUs69s4g-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-66fd88c39f6so7053256d6.0 for ; Mon, 30 Oct 2023 09:34:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683640; x=1699288440; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H7LWaJqNMU4MMH18+1IMpyA0GlKWGvM59R3vPm4Tplc=; b=QlvBkO/IbOiT0Smdvi3SDxzXCD0uxC3Sopu3fQW91DX/JBJ+R8GbaiguhKsaXiUj9E 0vrgjczXesG1ocxeSrVbhgwDaJu35skKlz4uu4Q2iYZN9yQq2Gaw4A2OSA6hAg76N4sc we0rdfUT8242r0X+RHyD33ib8Hjw3+3cG5lK77RARVT6G2NAnTqOFTodANHmfat2ZaEc RE73jq6cD/2BLmiVet+F2kuw18GqJ/Cq6FyYO8/fICZzHmoi0ueERrqZRK1ay1TG/nI7 Bm5gkINSu2TeQG4XGQMGiTDDzxVWy4oFY7QJqgBDUsRUhRtox6YuzEAwyb5S1FwNeJNL bcNQ== X-Gm-Message-State: AOJu0YzTcTEcqhcUhpF4mjiShW2IwajkMNhzwQlZ72tQJStslSp2pIVE 4pWRr0D1DfXwxelXxfGEYZK6IiYtqrGZFr9Ks8k6ZCThJM/aeC88S6SsX5QuM8lBP/HAs9BlqcM N0Kw/85r6WF9QJnMs0Ur3a1BHIsEfOPryHkUWP822T4km1fj0xMiT7v/eAa4I2jsqZvJEAhpl X-Received: by 2002:a05:622a:1a14:b0:412:2dac:acb7 with SMTP id f20-20020a05622a1a1400b004122dacacb7mr12559512qtb.1.1698683640512; Mon, 30 Oct 2023 09:34:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH9udNJfRKJwSAW1NpSYIPG3+xYNUlui17fjafoigBh2mVPyKhKXRgFYSBMsMSPOjdftYjNgA== X-Received: by 2002:a05:622a:1a14:b0:412:2dac:acb7 with SMTP id f20-20020a05622a1a1400b004122dacacb7mr12559482qtb.1.1698683640094; Mon, 30 Oct 2023 09:34:00 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id l15-20020ac84ccf000000b004198ae7f841sm3531111qtv.90.2023.10.30.09.33.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:33:59 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas , Joao Martins , Juan Quintela Subject: [PATCH v2 5/5] migration: Add tracepoints for downtime checkpoints Date: Mon, 30 Oct 2023 12:33:46 -0400 Message-ID: <20231030163346.765724-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231030163346.765724-1-peterx@redhat.com> References: <20231030163346.765724-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.483, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch is inspired by Joao Martin's patch here: https://lore.kernel.org/r/20230926161841.98464-1-joao.m.martins@oracle.com Add tracepoints for major downtime checkpoints on both src and dst. They share the same tracepoint with a string showing its stage. Besides the checkpoints in the previous patch, this patch also added destination checkpoints. On src, we have these checkpoints added: - src-downtime-start: right before vm stops on src - src-vm-stopped: after vm is fully stopped - src-iterable-saved: after all iterables saved (END sections) - src-non-iterable-saved: after all non-iterable saved (FULL sections) - src-downtime-stop: migration fully completed On dst, we have these checkpoints added: - dst-precopy-loadvm-completes: after loadvm all done for precopy - dst-precopy-bh-*: record BH steps to resume VM for precopy - dst-postcopy-bh-*: record BH steps to resume VM for postcopy On dst side, we don't have a good way to trace total time consumed by iterable or non-iterable for now. We can mark it by 1st time receiving a FULL / END section, but rather than that let's just rely on the other tracepoints added for vmstates to back up the information. With this patch, one can enable "vmstate_downtime*" tracepoints and it'll enable all tracepoints for downtime measurements necessary. Drop loadvm_postcopy_handle_run_bh() tracepoint alongside, because they service the same purpose, which was only for postcopy. We then have unified prefix for all downtime relevant tracepoints. Co-developed-by: Joao Martins Signed-off-by: Peter Xu Reviewed-by: Juan Quintela --- migration/migration.c | 16 +++++++++++++++- migration/savevm.c | 14 +++++++++----- migration/trace-events | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 9013c1b500..52f4ed41b2 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -103,6 +103,7 @@ static int close_return_path_on_source(MigrationState *s); static void migration_downtime_start(MigrationState *s) { + trace_vmstate_downtime_checkpoint("src-downtime-start"); s->downtime_start = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); } @@ -117,6 +118,8 @@ static void migration_downtime_end(MigrationState *s) if (!s->downtime) { s->downtime = now - s->downtime_start; } + + trace_vmstate_downtime_checkpoint("src-downtime-end"); } static bool migration_needs_multiple_sockets(void) @@ -151,7 +154,11 @@ static gint page_request_addr_cmp(gconstpointer ap, gconstpointer bp) int migration_stop_vm(RunState state) { - return vm_stop_force_state(state); + int ret = vm_stop_force_state(state); + + trace_vmstate_downtime_checkpoint("src-vm-stopped"); + + return ret; } void migration_object_init(void) @@ -500,6 +507,8 @@ static void process_incoming_migration_bh(void *opaque) Error *local_err = NULL; MigrationIncomingState *mis = opaque; + trace_vmstate_downtime_checkpoint("dst-precopy-bh-enter"); + /* If capability late_block_activate is set: * Only fire up the block code now if we're going to restart the * VM, else 'cont' will do it. @@ -525,6 +534,8 @@ static void process_incoming_migration_bh(void *opaque) */ qemu_announce_self(&mis->announce_timer, migrate_announce_params()); + trace_vmstate_downtime_checkpoint("dst-precopy-bh-announced"); + multifd_load_shutdown(); dirty_bitmap_mig_before_vm_start(); @@ -542,6 +553,7 @@ static void process_incoming_migration_bh(void *opaque) } else { runstate_set(global_state_get_runstate()); } + trace_vmstate_downtime_checkpoint("dst-precopy-bh-vm-started"); /* * This must happen after any state changes since as soon as an external * observer sees this event they might start to prod at the VM assuming @@ -576,6 +588,8 @@ process_incoming_migration_co(void *opaque) ret = qemu_loadvm_state(mis->from_src_file); mis->loadvm_co = NULL; + trace_vmstate_downtime_checkpoint("dst-precopy-loadvm-completed"); + ps = postcopy_state_get(); trace_process_incoming_migration_co_end(ret, ps); if (ps != POSTCOPY_INCOMING_NONE) { diff --git a/migration/savevm.c b/migration/savevm.c index cd6d6ba493..2578137ee7 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1494,6 +1494,8 @@ int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy) end_ts_each - start_ts_each); } + trace_vmstate_downtime_checkpoint("src-iterable-saved"); + return 0; } @@ -1560,6 +1562,8 @@ int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, json_writer_free(vmdesc); ms->vmdesc = NULL; + trace_vmstate_downtime_checkpoint("src-non-iterable-saved"); + return 0; } @@ -2102,18 +2106,18 @@ static void loadvm_postcopy_handle_run_bh(void *opaque) Error *local_err = NULL; MigrationIncomingState *mis = opaque; - trace_loadvm_postcopy_handle_run_bh("enter"); + trace_vmstate_downtime_checkpoint("dst-postcopy-bh-enter"); /* TODO we should move all of this lot into postcopy_ram.c or a shared code * in migration.c */ cpu_synchronize_all_post_init(); - trace_loadvm_postcopy_handle_run_bh("after cpu sync"); + trace_vmstate_downtime_checkpoint("dst-postcopy-bh-cpu-synced"); qemu_announce_self(&mis->announce_timer, migrate_announce_params()); - trace_loadvm_postcopy_handle_run_bh("after announce"); + trace_vmstate_downtime_checkpoint("dst-postcopy-bh-announced"); /* Make sure all file formats throw away their mutable metadata. * If we get an error here, just don't restart the VM yet. */ @@ -2124,7 +2128,7 @@ static void loadvm_postcopy_handle_run_bh(void *opaque) autostart = false; } - trace_loadvm_postcopy_handle_run_bh("after invalidate cache"); + trace_vmstate_downtime_checkpoint("dst-postcopy-bh-cache-invalidated"); dirty_bitmap_mig_before_vm_start(); @@ -2138,7 +2142,7 @@ static void loadvm_postcopy_handle_run_bh(void *opaque) qemu_bh_delete(mis->bh); - trace_loadvm_postcopy_handle_run_bh("return"); + trace_vmstate_downtime_checkpoint("dst-postcopy-bh-vm-started"); } /* After all discards we can start running and asking for pages */ diff --git a/migration/trace-events b/migration/trace-events index 5820add1f3..e54f317e3b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -17,7 +17,6 @@ loadvm_handle_recv_bitmap(char *s) "%s" loadvm_postcopy_handle_advise(void) "" loadvm_postcopy_handle_listen(const char *str) "%s" loadvm_postcopy_handle_run(void) "" -loadvm_postcopy_handle_run_bh(const char *str) "%s" loadvm_postcopy_handle_resume(void) "" loadvm_postcopy_ram_handle_discard(void) "" loadvm_postcopy_ram_handle_discard_end(void) "" @@ -50,6 +49,7 @@ vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_downtime_save(const char *type, const char *idstr, uint32_t instance_id, int64_t downtime) "type=%s idstr=%s instance_id=%d downtime=%"PRIi64 vmstate_downtime_load(const char *type, const char *idstr, uint32_t instance_id, int64_t downtime) "type=%s idstr=%s instance_id=%d downtime=%"PRIi64 +vmstate_downtime_checkpoint(const char *checkpoint) "%s" postcopy_pause_incoming(void) "" postcopy_pause_incoming_continued(void) "" postcopy_page_req_sync(void *host_addr) "sync page req %p"