From patchwork Wed Sep 21 21:43:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Stillson X-Patchwork-Id: 1680854 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=sZMjgSNt; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20210112.gappssmtp.com header.i=@rivosinc-com.20210112.gappssmtp.com header.a=rsa-sha256 header.s=20210112 header.b=4FXEAyWn; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MXsbd1QMHz1ypX for ; Thu, 22 Sep 2022 07:53:01 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zyiPEQ2uoWR0XhPrlDtxPDAU7nfISnvJvA7KRrWa+1I=; b=sZMjgSNtqpyW1v MXtu31Yrzs0a68ZC6vQL0pRr5jQZbV1AGr9/w79JI2jNl01rs5zCGXO0Pmqt8In6R8AR8iHSbsJfQ 79D0XTNiBa1ffFeENw/PXkpGH0p+A36UiHJQK9b1JQaUr0PO67bDZSsu/9Q3keflV0y3lcxhPKxV6 EDor659CbdkMH0yBfh6r4zHfYyjFj//SF0UYV2qqG2yvXWRgLc4osyOxO85ZXISEj4MNx6HafhVnd XL/XsolGD7gWB41Zup20Jo6V8UzZEB9gFCGX5EFo6sHg1bXXaTPLKWAtoW/4PA5VOUn4FqK/lARvK FWAIKraaSAopJSmBfJNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ob7eK-00Cmza-HR; Wed, 21 Sep 2022 21:52:56 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ob7eF-00Cmwd-Vw for kvm-riscv@lists.infradead.org; Wed, 21 Sep 2022 21:52:55 +0000 Received: by mail-pj1-x102c.google.com with SMTP id p1-20020a17090a2d8100b0020040a3f75eso167934pjd.4 for ; Wed, 21 Sep 2022 14:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=9QztRJhDFYW9WZfYeW38b+1O0y8hGYDxe0zbOFpNx/M=; b=4FXEAyWnl8rEPNR7wbfnj5FmGXdFG3ED7A5Trq3qXhO0TVRAkAzXiPi9m5aZu4TXdp 5gc4oJKyuy3QlWo8Zd3hGGQHg+9Z3RO3jY9PV5ln8dUqo9ty/PLDiMeFquJ8YxaLG8GP VAcPaec5zu3wKgRpoab3awGaBYP0XOZZmopsp0pGY/nhLfCGDz+A8GlBJ9+ZdvUoxqNr C+gdgHobTAA5WrJkfWII9pqdRzQ53hwmIbOPGQxSk8FK52ETaEDBp6llve5Su/T6KTWQ fPffBNJisYYv56YkayiFxA+AD2MUBbQbT+ttFlAwI9DLuIe6coi9VqGQ9XbGR30mhK3p LlSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=9QztRJhDFYW9WZfYeW38b+1O0y8hGYDxe0zbOFpNx/M=; b=1xXm3HeDHyg5KtFhQjS3rV4lsP7Vwk3MaSpfPzSx4C1T2AvW8NzN4QrFVbGlzDSruP ilgZjPd5dJwD1xV1EN0eXQgLu322QBTpOBLh/Kl3Y4YueoJkBu78QT8baTLpj+6JG5Jf 4OOyvxegzyLm12p0LWWTWHtXOWb0xR2zGipZrFymC8rX4qiAYn592jr0A1zPCf8+NWhO 7bylrbPhUfGvkNG5rupw5MyRLwcnRnZGjs0c53vJfgWQiN/gEbCf6oZ1TUQqSKKBpcfF DnG6pmhdvJ8gKRshOC+CH/sRCEsylFRUUmNsuBxPewzJOy6PLYyqYQjN+M2UI/aywO6f QczQ== X-Gm-Message-State: ACrzQf3wNcZpbMj+9+7NiPibix6adBeR6UPacfhAZlh0ORVgiDfFdGDP 3t+Gtmon8uaqVhmmHmi9nGfJpw== X-Google-Smtp-Source: AMsMyM7FtJQyPVFcN5hAmKNQxQ5f0hiJdHy7x3MFQ7yEuNRelMTlBhku+aHb3OLsUtpA4xwmLlO6Vw== X-Received: by 2002:a17:903:44e:b0:178:8492:d0ff with SMTP id iw14-20020a170903044e00b001788492d0ffmr126643plb.146.1663797170122; Wed, 21 Sep 2022 14:52:50 -0700 (PDT) Received: from stillson.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id k7-20020aa79727000000b005484d133127sm2634536pfg.129.2022.09.21.14.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 14:52:49 -0700 (PDT) From: Chris Stillson To: Cc: Greentime Hu , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , Eric Biederman , Kees Cook , Anup Patel , Atish Patra , Oleg Nesterov , Heinrich Schuchardt , Guo Ren , Chris Stillson , Conor Dooley , Paolo Bonzini , Alexandre Ghiti , Qinglin Pan , Arnd Bergmann , Heiko Stuebner , Jisheng Zhang , Dao Lu , "Peter Zijlstra (Intel)" , Sunil V L , Han-Kuan Chen , Li Zhengyu , Changbin Du , Alexander Graf , Ard Biesheuvel , Tsukasa OI , Yury Norov , Frederic Weisbecker , Mark Rutland , Nicolas Saenz Julienne , Vitaly Wool , Myrtle Shah , Nick Knight , Catalin Marinas , Will Deacon , Mark Brown , WANG Xuerui , Alexey Dobriyan , Huacai Chen , Christian Brauner , Eugene Syromiatnikov , Colin Cross , Peter Collingbourne , Andrew Morton , Suren Baghdasaryan , Barret Rhoden , Davidlohr Bueso , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org Subject: [PATCH v12 12/17] riscv: Add support for kernel mode vector Date: Wed, 21 Sep 2022 14:43:54 -0700 Message-Id: <20220921214439.1491510-12-stillson@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220921214439.1491510-1-stillson@rivosinc.com> References: <20220921214439.1491510-1-stillson@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220921_145252_051747_93847974 X-CRM114-Status: GOOD ( 21.99 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: From: Greentime Hu Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:102c listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org From: Greentime Hu Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu --- arch/riscv/include/asm/vector.h | 3 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 132 +++++++++++++++++++++++++ arch/riscv/kernel/vector.S | 9 ++ 4 files changed, 145 insertions(+) create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 16304b0c6a6f..a59841cc81fb 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -10,5 +10,8 @@ void rvv_enable(void); void rvv_disable(void); +void kernel_rvv_begin(void); +void kernel_rvv_end(void); +void vector_flush_cpu_state(void); #endif /* ! __ASM_RISCV_VECTOR_H */ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 35752fb6d145..8c238415f800 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -56,6 +56,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_VECTOR) += vector.o +obj-$(CONFIG_VECTOR) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..0277168af0c5 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include + +DECLARE_PER_CPU(bool, vector_context_busy); +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_vector - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_vector(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +/* + * kernel_rvv_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_vector() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_rvv_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_rvv_end() is + * called. + */ +void kernel_rvv_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + WARN_ON(!may_use_vector()); + + /* Acquire kernel mode vector */ + get_cpu_vector_context(); + + /* Save vector state, if any */ + vstate_save(current, task_pt_regs(current)); + + /* Enable vector */ + rvv_enable(); + + /* Invalidate vector regs */ + vector_flush_cpu_state(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_begin); + +/* + * kernel_rvv_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_rvv_begin() was previously + * called, with no call to kernel_rvv_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_rvv_begin() is called again in the meantime. + */ +void kernel_rvv_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + /* Invalidate vector regs */ + vector_flush_cpu_state(); + + /* Restore vector state, if any */ + vstate_restore(current, task_pt_regs(current)); + + /* disable vector */ + rvv_disable(); + + /* release kernel mode vector */ + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_end); diff --git a/arch/riscv/kernel/vector.S b/arch/riscv/kernel/vector.S index 9f7dc70c4443..9c2de823c0d9 100644 --- a/arch/riscv/kernel/vector.S +++ b/arch/riscv/kernel/vector.S @@ -91,3 +91,12 @@ ENTRY(rvv_disable) csrc CSR_STATUS, status ret ENDPROC(rvv_disable) + +ENTRY(vector_flush_cpu_state) + vsetvli t0, x0, e8, m8, ta, ma + vmv.v.i v0, 0 + vmv.v.i v8, 0 + vmv.v.i v16, 0 + vmv.v.i v24, 0 + ret +ENDPROC(vector_flush_cpu_state)