From patchwork Sun Nov 15 14:31:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zenghui Yu X-Patchwork-Id: 1400445 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=huawei.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CYvnG3jMCz9sSn for ; Mon, 16 Nov 2020 01:32:48 +1100 (AEDT) Received: from localhost ([::1]:35382 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1keJ5B-0003lg-Bs for incoming@patchwork.ozlabs.org; Sun, 15 Nov 2020 09:32:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55268) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1keJ4f-0003l2-3e; Sun, 15 Nov 2020 09:32:13 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:2854) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1keJ4W-0008CP-Kk; Sun, 15 Nov 2020 09:32:12 -0500 Received: from DGGEMS403-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4CYvlb2S1Zz15JTD; Sun, 15 Nov 2020 22:31:23 +0800 (CST) Received: from [10.174.185.179] (10.174.185.179) by DGGEMS403-HUB.china.huawei.com (10.3.19.203) with Microsoft SMTP Server id 14.3.487.0; Sun, 15 Nov 2020 22:31:28 +0800 From: Zenghui Yu Subject: [RFC] vfio-pci/migration: Dirty logging of the Memory BAR region? To: , Message-ID: Date: Sun, 15 Nov 2020 22:31:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 Content-Language: en-US X-Originating-IP: [10.174.185.179] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=yuzenghui@huawei.com; helo=szxga04-in.huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/15 09:31:36 X-ACL-Warn: Detected OS = Linux 3.1-3.10 [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Cornelia Huck , Kirti Wankhede , Alex Williamson , wanghaibin.wang@huawei.com, Paolo Bonzini , kvmarm@lists.cs.columbia.edu Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Hi folks, While trying the new vfio-pci migration on my arm64 server, I noticed an error at the very beginning: qemu-system-aarch64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=5, start=0x8000000000, size=0x100000: Invalid argument The reason is that we've registered the Memory BAR region as ram device region (mr->ram_device is set) and tried to track dirty pages of this region during migration. QEMU tries to request tracking of it (via kvm memory listener's log_start() callback) whilst KVM/arm64 clearly refuses to do so [1]: > int kvm_arch_prepare_memory_region(struct kvm *kvm, ...) > { > /* IO region dirty page logging not allowed */ > if (memslot->flags & KVM_MEM_LOG_DIRTY_PAGES) { > ret = -EINVAL; > goto out; > } > } If I understand things correctly, the Memory BAR region generally contains internal device-specific registers which shouldn't (and can't) be tracked by QEMU's dirty logging mechanism. I've patched QEMU with the following diff and it seems work for me, but all of these still require for comments. return mask; Thanks, Zenghui [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/mmu.c diff --git a/softmmu/memory.c b/softmmu/memory.c index 71951fe4dc..0958db1a08 100644 --- a/softmmu/memory.c +++ b/softmmu/memory.c @@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr) uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr) { uint8_t mask = mr->dirty_log_mask; - if (global_dirty_log && (mr->ram_block || memory_region_is_iommu(mr))) { + RAMBlock *rb = mr->ram_block; + + if (global_dirty_log && ((rb && qemu_ram_is_migratable(rb)) || + memory_region_is_iommu(mr))) { mask |= (1 << DIRTY_MEMORY_MIGRATION); }