From patchwork Fri Jan 11 06:37:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiao Guangrong X-Patchwork-Id: 1023407 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="fonA4IZN"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bY8z6rNcz9sCh for ; Fri, 11 Jan 2019 17:38:27 +1100 (AEDT) Received: from localhost ([127.0.0.1]:54291 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqSX-0002Ya-TC for incoming@patchwork.ozlabs.org; Fri, 11 Jan 2019 01:38:25 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57793) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqRy-0002X5-Lm for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghqRx-0007SD-PL for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:50 -0500 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]:36995) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghqRx-0007RU-JR for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:49 -0500 Received: by mail-pg1-x52d.google.com with SMTP id c25so5909811pgb.4 for ; Thu, 10 Jan 2019 22:37:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=YWB5ebsLtKqAkrxfbAsNN6bCEGxSWvVRBCXfnTjEnwo=; b=fonA4IZN9R77D7kiSQXX4501xsUSWWl8rWlqmqDjYi2wLvB/48QKIbdpJ8nLXYq8pw UEgLONebeKi6KnordW7u90j8ScyF5QyUXSfShDEUSJzLFbCJfRXjyoewVRjma/uKVOWo 8hlJtQ56gY8+xsQRglkhRf1U/OT4UG5Ujj7ES5sid2W4ZVENFeIyDENYXhYjQiI0/0/+ PLvuHoNhz+SpaLTmR/itMy9SbyDm1AYL0yfYk2Z/OOEB5+dCV14SF7V9Iy7ZUFQewGdd 9bzfdlPByBoIcKYqWaV2KiEjr1nKqSlpDnJJc9vgHg7JOgeeyYq9R91ZIWYvepsEK0Dp AO8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=YWB5ebsLtKqAkrxfbAsNN6bCEGxSWvVRBCXfnTjEnwo=; b=BIF0GBCHdQYTGEvKcZRoRVSoRK+hBNEctSQyek/ZJH9fXumOu0pyNymqlsaxxchlgV 8EDFS03Yws0LPr8wqyWGHdQuyX7cGSdoqJovSkR477hMesMqdj2U/hZyt+k5g09Mtn9z Xq7WYl3trrNA8WDRRSjPbVq+KPXoIUiqpJFjAdjmMAcreCIEioXh4Y8yi+XmT83E1AKu +2xYGzpmOg19/ZsMOXKRA3tinN9c5V/CHAFJ46Rk1B5l+ex1ISgzcmF7v+3GyUv7sW2S Rg8XJ1lecxXx4DebOxCO7Uz3lG8Azak8vtL5yqDEPSQwTgG+xgCoJwZfc1uAjghq8rs5 WfLw== X-Gm-Message-State: AJcUukesgqceBzHFaG1CJTfWNvmElowAtpxq06v1P7h9uAea1zSzyms7 l9o0sVWDAInqIKTYuHzw65M= X-Google-Smtp-Source: ALg8bN4bxri16pzC/F9hK/sQ+AS7mw1PYv+Ow1hw/fnzS5X4WkXiG0cxGTSKEaPrYBd0ISSY7+mRuQ== X-Received: by 2002:a63:f34b:: with SMTP id t11mr12308569pgj.341.1547188667872; Thu, 10 Jan 2019 22:37:47 -0800 (PST) Received: from localhost.localdomain ([203.205.141.36]) by smtp.gmail.com with ESMTPSA id 78sm141460933pft.184.2019.01.10.22.37.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 10 Jan 2019 22:37:47 -0800 (PST) From: guangrong.xiao@gmail.com X-Google-Original-From: xiaoguangrong@tencent.com To: pbonzini@redhat.com, mst@redhat.com, mtosatti@redhat.com Date: Fri, 11 Jan 2019 14:37:29 +0800 Message-Id: <20190111063732.10484-1-xiaoguangrong@tencent.com> X-Mailer: git-send-email 2.14.5 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52d Subject: [Qemu-devel] [PATCH v2 0/3] optimize waiting for free thread to do compression X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, quintela@redhat.com, Xiao Guangrong , qemu-devel@nongnu.org, peterx@redhat.com, dgilbert@redhat.com, wei.w.wang@intel.com, cota@braap.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Xiao Guangrong Changelog in v2: squash 'compress-wait-thread-adaptive' into 'compress-wait-thread' based on peter's suggestion Currently we have two behaviors if all threads are busy to do compression, the main thread mush wait one of them becoming free if @compress-wait-thread set to on or the main thread can directly return without wait and post the page out as normal one Both of them have its profits and short-comes, however, if the bandwidth is not limited extremely so that compression can not use out all of it bandwidth, at the same time, the migration process is easily throttled if we posted too may pages as normal pages. None of them can work properly under this case In order to use the bandwidth more effectively, we introduce the third behavior, adaptive, which make the main thread wait if there is no bandwidth left or let the page go out as normal page if there has enough bandwidth to make sure the migration process will not be throttled Another patch introduces a new statistic, pages-per-second, as bandwidth or mbps is not enough to measure the performance of posting pages out as we have compression, xbzrle, which can significantly reduce the amount of the data size, instead, pages-per-second is the one we want Performance data ================ We have limited the bandwidth to 300 Used Bandwidth Pages-per-Second compress-wait-thread = on 951.66 mbps 131784 compress-wait-thread = off 2491.74 mbps 93495 compress-wait-thread-adaptive 1982.94 mbps 162529 = on Xiao Guangrong (3): migration: introduce pages-per-second migration: fix memory leak when updating tls-creds and tls-hostname migration: introduce adaptive model for waiting thread hmp.c | 8 ++- migration/migration.c | 137 +++++++++++++++++++++++++++++++++++++++++--------- migration/migration.h | 21 ++++++++ migration/ram.c | 122 ++++++++++++++++++++++++++++++++++++++++---- qapi/migration.json | 44 +++++++++++----- 5 files changed, 281 insertions(+), 51 deletions(-)