From patchwork Thu Nov 26 08:28:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Vivier X-Patchwork-Id: 1406459 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4ChWDS174dz9sTv for ; Thu, 26 Nov 2020 19:30:48 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JS48FU4r; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JS48FU4r; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4ChWDR0nB1zDr7d for ; Thu, 26 Nov 2020 19:30:47 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=63.128.21.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=lvivier@redhat.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JS48FU4r; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JS48FU4r; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4ChWBc4LXLzDqvy for ; Thu, 26 Nov 2020 19:29:11 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606379348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3rXQzmEocIXb9J6qVfLPhdebmYZHUsNEtxEoa/mad+8=; b=JS48FU4r25PUzsR61DQbEXzDssFAqcRRY9Vx/Vqtkd+dmROW11g4b3iMFsbSWuV/Vl5UQw ZW5g2jrqd2KUURl8D+cHkxmlra121iVoycP6MKEJtxtxw8rFruHzaMcW3FnYJRgkY1IAVo E2dTTLdHC1ru9+WQ8B9VVlxSXajLyYQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606379348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3rXQzmEocIXb9J6qVfLPhdebmYZHUsNEtxEoa/mad+8=; b=JS48FU4r25PUzsR61DQbEXzDssFAqcRRY9Vx/Vqtkd+dmROW11g4b3iMFsbSWuV/Vl5UQw ZW5g2jrqd2KUURl8D+cHkxmlra121iVoycP6MKEJtxtxw8rFruHzaMcW3FnYJRgkY1IAVo E2dTTLdHC1ru9+WQ8B9VVlxSXajLyYQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-320-gV7_s-NHP3urQRXK5900yQ-1; Thu, 26 Nov 2020 03:29:03 -0500 X-MC-Unique: gV7_s-NHP3urQRXK5900yQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 56C2181CBE1; Thu, 26 Nov 2020 08:29:01 +0000 (UTC) Received: from thinkpad.redhat.com (ovpn-113-83.ams2.redhat.com [10.36.113.83]) by smtp.corp.redhat.com (Postfix) with ESMTP id 106CF5C1B4; Thu, 26 Nov 2020 08:28:53 +0000 (UTC) From: Laurent Vivier To: linux-kernel@vger.kernel.org Subject: [PATCH v4 0/2] powerpc/pseries: fix MSI/X IRQ affinity on pseries Date: Thu, 26 Nov 2020 09:28:50 +0100 Message-Id: <20201126082852.1178497-1-lvivier@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Laurent Vivier , "Michael S . Tsirkin" , linux-pci@vger.kernel.org, Greg Kurz , linux-block@vger.kernel.org, Paul Mackerras , Marc Zyngier , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christoph Hellwig Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" With virtio, in multiqueue case, each queue IRQ is normally bound to a different CPU using the affinity mask. This works fine on x86_64 but totally ignored on pseries. This is not obvious at first look because irqbalance is doing some balancing to improve that. It appears that the "managed" flag set in the MSI entry is never copied to the system IRQ entry. This series passes the affinity mask from rtas_setup_msi_irqs() to irq_domain_alloc_descs() by adding an affinity parameter to irq_create_mapping(). The first patch adds the parameter (no functional change), the second patch passes the actual affinity mask to irq_create_mapping() in rtas_setup_msi_irqs(). For instance, with 32 CPUs VM and 32 queues virtio-scsi interface: ... -smp 32 -device virtio-scsi-pci,id=virtio_scsi_pci0,num_queues=32 for IRQ in $(grep virtio2-request /proc/interrupts |cut -d: -f1); do for file in /proc/irq/$IRQ/ ; do echo -n "IRQ: $(basename $file) CPU: " ; cat $file/smp_affinity_list done done Without the patch (and without irqbalanced) IRQ: 268 CPU: 0-31 IRQ: 269 CPU: 0-31 IRQ: 270 CPU: 0-31 IRQ: 271 CPU: 0-31 IRQ: 272 CPU: 0-31 IRQ: 273 CPU: 0-31 IRQ: 274 CPU: 0-31 IRQ: 275 CPU: 0-31 IRQ: 276 CPU: 0-31 IRQ: 277 CPU: 0-31 IRQ: 278 CPU: 0-31 IRQ: 279 CPU: 0-31 IRQ: 280 CPU: 0-31 IRQ: 281 CPU: 0-31 IRQ: 282 CPU: 0-31 IRQ: 283 CPU: 0-31 IRQ: 284 CPU: 0-31 IRQ: 285 CPU: 0-31 IRQ: 286 CPU: 0-31 IRQ: 287 CPU: 0-31 IRQ: 288 CPU: 0-31 IRQ: 289 CPU: 0-31 IRQ: 290 CPU: 0-31 IRQ: 291 CPU: 0-31 IRQ: 292 CPU: 0-31 IRQ: 293 CPU: 0-31 IRQ: 294 CPU: 0-31 IRQ: 295 CPU: 0-31 IRQ: 296 CPU: 0-31 IRQ: 297 CPU: 0-31 IRQ: 298 CPU: 0-31 IRQ: 299 CPU: 0-31 With the patch: IRQ: 265 CPU: 0 IRQ: 266 CPU: 1 IRQ: 267 CPU: 2 IRQ: 268 CPU: 3 IRQ: 269 CPU: 4 IRQ: 270 CPU: 5 IRQ: 271 CPU: 6 IRQ: 272 CPU: 7 IRQ: 273 CPU: 8 IRQ: 274 CPU: 9 IRQ: 275 CPU: 10 IRQ: 276 CPU: 11 IRQ: 277 CPU: 12 IRQ: 278 CPU: 13 IRQ: 279 CPU: 14 IRQ: 280 CPU: 15 IRQ: 281 CPU: 16 IRQ: 282 CPU: 17 IRQ: 283 CPU: 18 IRQ: 284 CPU: 19 IRQ: 285 CPU: 20 IRQ: 286 CPU: 21 IRQ: 287 CPU: 22 IRQ: 288 CPU: 23 IRQ: 289 CPU: 24 IRQ: 290 CPU: 25 IRQ: 291 CPU: 26 IRQ: 292 CPU: 27 IRQ: 293 CPU: 28 IRQ: 294 CPU: 29 IRQ: 295 CPU: 30 IRQ: 299 CPU: 31 This matches what we have on an x86_64 system. v4: udate changelog of PATCH 2, add Michael's Acked-by v3: update changelog of PATCH 1 with comments from Thomas Gleixner and Marc Zyngier. v2: add a wrapper around original irq_create_mapping() with the affinity parameter. Update comments Laurent Vivier (2): genirq/irqdomain: Add an irq_create_mapping_affinity() function powerpc/pseries: Pass MSI affinity to irq_create_mapping() arch/powerpc/platforms/pseries/msi.c | 3 ++- include/linux/irqdomain.h | 12 ++++++++++-- kernel/irq/irqdomain.c | 13 ++++++++----- 3 files changed, 20 insertions(+), 8 deletions(-)