From patchwork Sat Feb 23 14:16:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1047394 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="TPi/d/Mw"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4469KC4m8nz9s71 for ; Sun, 24 Feb 2019 01:17:51 +1100 (AEDT) Received: from localhost ([127.0.0.1]:38510 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gxY7h-0006UR-Fo for incoming@patchwork.ozlabs.org; Sat, 23 Feb 2019 09:17:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42446) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gxY6r-0006TI-BI for qemu-devel@nongnu.org; Sat, 23 Feb 2019 09:16:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gxY6p-0002ND-M6 for qemu-devel@nongnu.org; Sat, 23 Feb 2019 09:16:57 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:52128) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gxY6p-0002Mw-DC for qemu-devel@nongnu.org; Sat, 23 Feb 2019 09:16:55 -0500 Received: by mail-wm1-x342.google.com with SMTP id n19so4326041wmi.1 for ; Sat, 23 Feb 2019 06:16:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uoe/T0o0qHqeHNH63zoGnv6wP3JeS+alu3KSVxaJfO8=; b=TPi/d/MwY1yw05j3yrB5W8RjgSZLxZRb0FgtnGZg2xfN6Vq4rQWy2/nq4owmJ8Kj8T 9Kj/OY+wYJFXWQuLo6YYti8LDLPLYrrYfMLG7qZcp+koj1Ik1so/gynWBFfDMmi48eQt Qe/fyMDvnslsapVeorWdvtQpLytjeVKOdrw45Pi09Sy5fdT5UpsVYRNqX/aE7rORbLw/ +vj6lMwtUs8t3lYJgnjyjFLc2O1dJHNiSDhQirhq1Bj8nqPhowvaA6XmoH/obyNWL4nB yWkqcPS2tVMibg3stGgEevzSS77+kZM5WGgRgWcN1lg9ejQgMDWuq3nJqvQbUcx8RLLt SobA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uoe/T0o0qHqeHNH63zoGnv6wP3JeS+alu3KSVxaJfO8=; b=YkOcmQKE6ddivVs5isgXi0HoXwF+tbTEpSJRdUCm29PQLiJQFM5cMebzvM8dLhab0B 4FC70MSbJUMuecw4vKQsKqwDABbSkE6ZT16YeK4A1EoGigILZZQq+c2pNNVJ80EKLAua 96dU+OaCD6ntq3TBuuRGrduUBzcsGy2aAyZ8ccmu+LGgqDJFgmW5F5y0cCxQ6C3kx0mL /gn5l6mP6//4aJnH91CxnXZEQvP1FU37phgOlhD0xMVEp0q+yCV8JbzDbijM4r9Jnf0+ seToujKmZ018wZ3ZPKaoEAuSlovxNLkaQiu9f1+EsVi70FY//yiRiYVk2GoWG94DR9oL iMtg== X-Gm-Message-State: AHQUAubRjHljz4gGHdkm9RsQ671B4LBGs9ACEc2J8ZgkKrRv0DXP/UME 0GLWqr2E/Ac050SbfZWKUtuGwe8d X-Google-Smtp-Source: AHgI3IYnZZmI2XQL7jZgiWWixM8sDc9EfpiLbGEljaujX7lpYDJS/GvLbxESPazJ+dypQqkYJhxBWg== X-Received: by 2002:a7b:c7c1:: with SMTP id z1mr5399519wmk.77.1550931413495; Sat, 23 Feb 2019 06:16:53 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id x3sm3310506wru.76.2019.02.23.06.16.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 23 Feb 2019 06:16:52 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 23 Feb 2019 16:16:36 +0200 Message-Id: <20190223141644.6609-4-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190223141644.6609-1-marcel.apfelbaum@gmail.com> References: <20190223141644.6609-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH PULL 03/11] hw/rdma: Introduce protected qlist X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, kamalheib1@gmail.com, yuval.shaia@oracle.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia To make code more readable move handling of protected list to a rdma_utils Signed-off-by: Yuval Shaia Message-Id: <20190213065357.16076-4-yuval.shaia@oracle.com> Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 20 +++++-------------- hw/rdma/rdma_backend_defs.h | 8 ++------ hw/rdma/rdma_utils.c | 39 +++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_utils.h | 9 +++++++++ 4 files changed, 55 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 5f60856d19..6e9c4617da 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -527,9 +527,7 @@ static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, bctx->up_ctx = ctx; bctx->sge = *sge; - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + rdma_protected_qlist_append_int64(&backend_dev->recv_mads_list, bctx_id); return 0; } @@ -913,23 +911,19 @@ static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, static void process_incoming_mad_req(RdmaBackendDev *backend_dev, RdmaCmMuxMsg *msg) { - QObject *o_ctx_id; unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; trace_mad_message("recv", msg->umad.mad, msg->umad_len); - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - o_ctx_id = qlist_pop(backend_dev->recv_mads_list.list); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); - if (!o_ctx_id) { + cqe_ctx_id = rdma_protected_qlist_pop_int64(&backend_dev->recv_mads_list); + if (cqe_ctx_id == -ENOENT) { rdma_warn_report("No more free MADs buffers, waiting for a while"); sleep(THR_POLL_TO); return; } - cqe_ctx_id = qnum_get_uint(qobject_to(QNum, o_ctx_id)); bctx = rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); if (unlikely(!bctx)) { rdma_error_report("No matching ctx for req %ld", cqe_ctx_id); @@ -994,8 +988,7 @@ static int mad_init(RdmaBackendDev *backend_dev, CharBackend *mad_chr_be) return -EIO; } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); - backend_dev->recv_mads_list.list = qlist_new(); + rdma_protected_qlist_init(&backend_dev->recv_mads_list); enable_rdmacm_mux_async(backend_dev); @@ -1010,10 +1003,7 @@ static void mad_fini(RdmaBackendDev *backend_dev) { disable_rdmacm_mux_async(backend_dev); qemu_chr_fe_disconnect(backend_dev->rdmacm_mux.chr_be); - if (backend_dev->recv_mads_list.list) { - qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); - qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); - } + rdma_protected_qlist_destroy(&backend_dev->recv_mads_list); } int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 15ae8b970e..a8c15b09ab 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -20,6 +20,7 @@ #include "chardev/char-fe.h" #include #include "contrib/rdmacm-mux/rdmacm-mux.h" +#include "rdma_utils.h" typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -30,11 +31,6 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; -typedef struct RecvMadList { - QemuMutex lock; - QList *list; -} RecvMadList; - typedef struct RdmaCmMux { CharBackend *chr_be; int can_receive; @@ -48,7 +44,7 @@ typedef struct RdmaBackendDev { struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - RecvMadList recv_mads_list; + RdmaProtectedQList recv_mads_list; RdmaCmMux rdmacm_mux; } RdmaBackendDev; diff --git a/hw/rdma/rdma_utils.c b/hw/rdma/rdma_utils.c index f1c980c6be..672a09079a 100644 --- a/hw/rdma/rdma_utils.c +++ b/hw/rdma/rdma_utils.c @@ -14,6 +14,8 @@ */ #include "qemu/osdep.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" #include "trace.h" #include "rdma_utils.h" @@ -55,3 +57,40 @@ void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len) pci_dma_unmap(dev, buffer, len, DMA_DIRECTION_TO_DEVICE, 0); } } + +void rdma_protected_qlist_init(RdmaProtectedQList *list) +{ + qemu_mutex_init(&list->lock); + list->list = qlist_new(); +} + +void rdma_protected_qlist_destroy(RdmaProtectedQList *list) +{ + if (list->list) { + qlist_destroy_obj(QOBJECT(list->list)); + qemu_mutex_destroy(&list->lock); + list->list = NULL; + } +} + +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value) +{ + qemu_mutex_lock(&list->lock); + qlist_append_int(list->list, value); + qemu_mutex_unlock(&list->lock); +} + +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list) +{ + QObject *obj; + + qemu_mutex_lock(&list->lock); + obj = qlist_pop(list->list); + qemu_mutex_unlock(&list->lock); + + if (!obj) { + return -ENOENT; + } + + return qnum_get_uint(qobject_to(QNum, obj)); +} diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index acd148837f..a8bf1d4fec 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -29,8 +29,17 @@ #define rdma_info_report(fmt, ...) \ info_report("%s: " fmt, "rdma", ## __VA_ARGS__) +typedef struct RdmaProtectedQList { + QemuMutex lock; + QList *list; +} RdmaProtectedQList; + void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); +void rdma_protected_qlist_init(RdmaProtectedQList *list); +void rdma_protected_qlist_destroy(RdmaProtectedQList *list); +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value); +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list); static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) {