From patchwork Wed Feb 27 19:19:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guilherme G. Piccoli" X-Patchwork-Id: 1049118 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 448lr852jSz9sBL; Thu, 28 Feb 2019 06:20:06 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1gz4kI-0005Md-9w; Wed, 27 Feb 2019 19:19:58 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.86_2) (envelope-from ) id 1gz4kG-0005MQ-CV for kernel-team@lists.ubuntu.com; Wed, 27 Feb 2019 19:19:56 +0000 Received: from mail-qt1-f200.google.com ([209.85.160.200]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1gz4kG-0008IW-2k for kernel-team@lists.ubuntu.com; Wed, 27 Feb 2019 19:19:56 +0000 Received: by mail-qt1-f200.google.com with SMTP id q17so16092828qta.17 for ; Wed, 27 Feb 2019 11:19:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=iFshU1eZHaz7r5EhCY7ouWKBok9ZJmtzHGECWmru1Ss=; b=KAldRUEpBLtGKxnJjEufnMi1gxTwLF1BeGR6OGu07xxf05XGScdCscdJUGLi4Fyh05 LVYsbuTyZMmF1zY7E7GXrLsTJWuOUbQrTIFufJvJ8DDeZMgJLkH1iOdG6X1K1rvv60jy Jz8xVQieIwbCbdBR1hsxlPUk1YvSPOK9KQV17JTfXMvAwChCiTxDfUT22kh7evfuj2ab gvznMtMRyq8B2N99YJoNSrGGJu8JKp5ID/vm4h9ZKBTfz9m8mL+FitO28TzEYgM39Bu7 binhQ1kIGFUoXvSFTtjWU13/9y0STjXa1uefmJQ0VFzW2laP2Eh5vVy+pMieCJjoSykg NydA== X-Gm-Message-State: APjAAAWBq3ClBIrXD1LHs6L72laiBuljKeDHSC41980YSIPv82EA6A6p mxrhWNChN2PgnKDMJNLmq/lwhivrGrRZTB51nYDudUuKbsZsneWYWxmg2RF8WSqfoKL0mr3Ic8i loTtRqRdqBucNowCy/tqbovHRDMZQ4g64RQEhJs+ogg== X-Received: by 2002:a0c:aec8:: with SMTP id n8mr3246801qvd.196.1551295194915; Wed, 27 Feb 2019 11:19:54 -0800 (PST) X-Google-Smtp-Source: APXvYqx3spXdBtqNmgc/bWnr9JGHuCbmUJ9VPJ0KDTgVDoov1ZrXBOfisBz/k0/5apRmLAw1t8INGA== X-Received: by 2002:a0c:aec8:: with SMTP id n8mr3246790qvd.196.1551295194700; Wed, 27 Feb 2019 11:19:54 -0800 (PST) Received: from localhost ([179.110.167.246]) by smtp.gmail.com with ESMTPSA id x25sm3882706qtx.71.2019.02.27.11.19.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Feb 2019 11:19:53 -0800 (PST) From: "Guilherme G. Piccoli" To: kernel-team@lists.ubuntu.com Subject: [SRU X] [PATCH 0/2] Hard lockups due to unrestricted lapic timer delay Date: Wed, 27 Feb 2019 16:19:48 -0300 Message-Id: <20190227191950.12074-1-gpiccoli@canonical.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gpiccoli@canonical.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/1817918 [Impact] * There is a long-time report of an issue with the TSC delay present in wait_lapic_expire() - basically the guest could have an expiration timer configured in a way it induces host to wait a long time (with preemption disabled), so there's a potential scenario for host lockups. * The stack trace we have access (from an user report of this issue) is (summarized) below: NMI watchdog: Watchdog detected hard LOCKUP on cpu 16 [...] CPU: 16 PID: 3024910 Comm: CPU 0/KVM Not tainted 4.4.0-139-generic #165-Ubuntu RIP: 0010:[] [] delay_tsc+0x20/0x60 [...] __delay+0x15/0x20 wait_lapic_expire+0xc3/0x150 [kvm] vcpu_enter_guest+0x743/0x11d0 [kvm] kvm_arch_vcpu_ioctl_run+0xe6/0x410 [kvm] kvm_vcpu_ioctl+0x33d/0x620 [kvm] do_vfs_ioctl+0x2af/0x4b0 ? __do_page_fault+0x1c1/0x410 ? fire_user_return_notifiers+0x3e/0x50 SyS_ioctl+0x79/0x90 entry_SYSCALL_64_fastpath+0x22/0xc1 This matches the reported problem in the KVM mailing-list: https://marc.info/?l=kvm&m=146374488028339 * A fix was proposed in the above thread, but discarded in favor of the following approach: https://marc.info/?l=kvm&m=146647260109315 The patch was merged in Linus tree, hence we hereby request the SRU: b606f189c7d5 ("KVM: LAPIC: cap __delay at lapic_timer_advance_ns"). There's one additional patch needed, which is just the header adjustment for exporting a necessary function. * The patch is missing only in 4.4 kernel series; Bionic (4.15) and the other * newer releases have the patch already. [Test Case] * Unfortunately this is a hard to reproduce issue; we have reports of this lockup from an user, hence the SRU request here. Also, the patch was introduced originally in kernel 4.7, approx. 2.5 years ago. So, we are confident that community is running this code long enough without errors reported. Also, checked in the Linus tree and no fixes for this code were introduced since kernel 4.7. [Regression Potential] * The code modification requested here affects the amount of delay in a specific timer; the patch introduces a maximum time for delay, preventing unbounded delays in host. The regression potential is considered low, and given the nature of the modification, latency issues in guests are likely to be the most problematic regression potential we have. Marcelo Tosatti (2): KVM: x86: move nsec_to_cycles from x86.c to x86.h KVM: LAPIC: cap __delay at lapic_timer_advance_ns arch/x86/kvm/lapic.c | 3 ++- arch/x86/kvm/x86.c | 6 ------ arch/x86/kvm/x86.h | 8 ++++++++ 3 files changed, 10 insertions(+), 7 deletions(-) Acked-by: Marcelo Henrique Cerri