From patchwork Tue Jun 18 05:28:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Ruffell X-Patchwork-Id: 1948940 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W3FfN1KDVz20Wb for ; Tue, 18 Jun 2024 15:28:43 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1sJROO-0005ri-0p; Tue, 18 Jun 2024 05:28:28 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1sJRON-0005rb-8s for kernel-team@lists.ubuntu.com; Tue, 18 Jun 2024 05:28:27 +0000 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 101B93F2B2 for ; Tue, 18 Jun 2024 05:28:27 +0000 (UTC) Received: by mail-pf1-f199.google.com with SMTP id d2e1a72fcca58-705bbf10fa1so4562378b3a.0 for ; Mon, 17 Jun 2024 22:28:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718688505; x=1719293305; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BLGCJU0jhheyNe8Txa0yfNYDCoDdrIUgBt1O96At2no=; b=SvWYZ8Qk+vOwT5LLTtNDKJx7Via+AJ95beYSkVIyd1SNryzD7ilRZSfba28hxjrY+x HuJ66xptlfeOqOgg7mCZqUwEN6Bf5YDU+sQd0LpkB0MT6ZawnL3ZD4TIHhw5Z2wQsytM SZI7ICQgc5sngv4+APi+xXI4gO6N830YTur4/NE7+ICSPFYDxQ/VkS54xilU+FbF2H6f tyErgZy/p0qmTugZzbyCMZklk7q7AcS6nY7phXp3/8JRyqcS5LYq32ZO3IXVw6wnvgpm hXDTTAJYKSE4A3/TnnnloiZLrbfxk90LPsK/ewtfBE14/kSB/HEm7k41XefZnJvRCp0B VnGA== X-Gm-Message-State: AOJu0YyhMwqPdvaqAmtoNsqvnts5z8koZhE9uEnsHeBRe+vEI24KIgJ3 Vza4MpBRCO9xq5R9ozCUT8DBxYt79/0U/gTEsHBsowIqsek9jQ7XCmKwhgBa0eltENMu7LYFTqm SSpB3oqk9mgW2ZPBXLf2qYh8xotESIF6JnrtF0I0U7p5j4hwBasg8/5er5Svdu/jjcB2sO9ZfCj ScpPAZvMhpMQ== X-Received: by 2002:a05:6a00:9282:b0:705:ddd4:8491 with SMTP id d2e1a72fcca58-705ddd48b01mr11608757b3a.0.1718688505329; Mon, 17 Jun 2024 22:28:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEEbLzRrsu+XXy3WXpY8LRNyTkbJfyt1D2wk8v+TTQMeGuLm3xdyhlJWEe35WOrtd7fv7z1xA== X-Received: by 2002:a05:6a00:9282:b0:705:ddd4:8491 with SMTP id d2e1a72fcca58-705ddd48b01mr11608747b3a.0.1718688504727; Mon, 17 Jun 2024 22:28:24 -0700 (PDT) Received: from ThinkPad-X1.. (222-154-76-179-fibre.sparkbb.co.nz. [222.154.76.179]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-705ccb91b62sm8196865b3a.203.2024.06.17.22.28.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 22:28:24 -0700 (PDT) From: Matthew Ruffell To: kernel-team@lists.ubuntu.com Subject: [SRU][Noble][PATCH 0/1] Removing legacy virtio-pci devices causes kernel panic Date: Tue, 18 Jun 2024 17:28:17 +1200 Message-Id: <20240618052818.38993-1-matthew.ruffell@canonical.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2067862 [Impact] If you detach a legacy virtio-pci device from a current Noble system, it will cause a null pointer dereference, and panic the system. This is an issue if you force noble to use legacy virtio-pci devices, or run noble on very old hypervisors that only support legacy virtio-pci devices, e.g. trusty and older. BUG: kernel NULL pointer dereference, address: 0000000000000000 ... CPU: 2 PID: 358 Comm: kworker/u8:3 Kdump: loaded Not tainted 6.8.0-31-generic #31-Ubuntu Workqueue: kacpi_hotplug acpi_hotplug_work_fn RIP: 0010:0x0 ... Call Trace: ? show_regs+0x6d/0x80 ? __die+0x24/0x80 ? page_fault_oops+0x99/0x1b0 ? do_user_addr_fault+0x2ee/0x6b0 ? exc_page_fault+0x83/0x1b0 ? asm_exc_page_fault+0x27/0x30 vp_del_vqs+0x6e/0x2a0 remove_vq_common+0x166/0x1a0 virtnet_remove+0x61/0x80 virtio_dev_remove+0x3f/0xc0 device_remove+0x40/0x80 device_release_driver_internal+0x20b/0x270 device_release_driver+0x12/0x20 bus_remove_device+0xcb/0x140 device_del+0x161/0x3e0 ? pci_bus_generic_read_dev_vendor_id+0x2c/0x1a0 device_unregister+0x17/0x60 unregister_virtio_device+0x16/0x40 virtio_pci_remove+0x43/0xa0 pci_device_remove+0x36/0xb0 device_remove+0x40/0x80 device_release_driver_internal+0x20b/0x270 device_release_driver+0x12/0x20 pci_stop_bus_device+0x7a/0xb0 pci_stop_and_remove_bus_device+0x12/0x30 disable_slot+0x4f/0xa0 acpiphp_disable_and_eject_slot+0x1c/0xa0 hotplug_event+0x11b/0x280 ? __pfx_acpiphp_hotplug_notify+0x10/0x10 acpiphp_hotplug_notify+0x27/0x70 acpi_device_hotplug+0xb6/0x300 acpi_hotplug_work_fn+0x1e/0x40 process_one_work+0x16c/0x350 worker_thread+0x306/0x440 ? _raw_spin_lock_irqsave+0xe/0x20 ? __pfx_worker_thread+0x10/0x10 kthread+0xef/0x120 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x44/0x70 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 The issue was introduced in: commit fd27ef6b44bec26915c5b2b22c13856d9f0ba17a Author: Feng Liu Date: Tue Dec 19 11:32:40 2023 +0200 Subject: virtio-pci: Introduce admin virtqueue Link: https://github.com/torvalds/linux/commit/fd27ef6b44bec26915c5b2b22c13856d9f0ba17a Modern virtio-pci devices are not affected. If the device is a legacy virtio device, the is_avq function pointer is not assigned in the virtio_pci_device structure of the legacy virtio device, resulting in a NULL pointer dereference when the code calls if (vp_dev->is_avq(vdev, vq->index)). There is no workaround. If you are affected, then not detaching devices for the time being is the only solution. [Fix] This was fixed in 6.9-rc1 by: commit c8fae27d141a32a1624d0d0d5419d94252824498 From: Li Zhang Date: Sat, 16 Mar 2024 13:25:54 +0800 Subject: virtio-pci: Check if is_avq is NULL Link: https://github.com/torvalds/linux/commit/c8fae27d141a32a1624d0d0d5419d94252824498 This is a clean cherry pick to noble. The commit just adds a basic NULL pointer check before it dereferences the pointer. [Testcase] Start a fresh Noble VM. Edit the grub kernel command line: 1) sudo vim /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="virtio_pci.force_legacy=1" 2) sudo update-grub 3) sudo reboot Outside the VM, on the host: $ qemu-img create -f qcow2 /root/share-device.qcow2 2G $ cat >> share-device.xml << EOF disk type='file' device='disk'> EOF $ sudo -s # virsh attach-device noble-test share-device.xml --config --live # virsh detach-device noble-test share-device.xml --config --live A kernel panic should occur. There is a test kernel available in: https://launchpad.net/~mruffell/+archive/ubuntu/lp2067862-test If you install it, the panic should no longer occur. [Where problems could occur] We are adding a basic null pointer check right before the pointer is about to be used, which is quite low risk. If a regression were to occur, it would only affect VMs using legacy virtio-pci devices, which is not the default. It would potentially have large impacts on fleets of very old hypervisors running trusty, precise or lucid, but that is very unlikely in this day and age. [Other Info] Upstream mailing list discussion and author testcase: https://lore.kernel.org/kvm/CACGkMEs1t-ipP7TasHkKNKd=peVEES6Xdw1zSsJkb-bc9Etx9Q@mail.gmail.com/T/#m167335bf7ab09b12fec3bdc5d46a30bc2e26cac7 Li Zhang (1): virtio-pci: Check if is_avq is NULL drivers/virtio/virtio_pci_common.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Acked-by: Paolo Pisati Acked-by: Manuel Diewald Acked-by: Chris Chiu