From patchwork Sat Mar 16 14:34:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1057366 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EPevzEi7"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44M4jz09sbz9s00 for ; Sun, 17 Mar 2019 01:35:34 +1100 (AEDT) Received: from localhost ([127.0.0.1]:42541 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h5APM-0000pW-Lf for incoming@patchwork.ozlabs.org; Sat, 16 Mar 2019 10:35:32 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49550) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h5AOR-0000oD-FL for qemu-devel@nongnu.org; Sat, 16 Mar 2019 10:34:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h5AOP-0000Wm-51 for qemu-devel@nongnu.org; Sat, 16 Mar 2019 10:34:35 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:39366) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1h5AOO-0000Vt-Kh for qemu-devel@nongnu.org; Sat, 16 Mar 2019 10:34:33 -0400 Received: by mail-wr1-x441.google.com with SMTP id p8so12457938wrq.6 for ; Sat, 16 Mar 2019 07:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3BHdWRw/I2mQJ5ddyQvsNmHD93DWER8WHM1dpXT9UjU=; b=EPevzEi7rSUDG2L3nW0bvq+gvrO783D+NRh8g0KGPjtK4Wal0urcKfqOVM1mayb5d0 0bK/uF3cEEG6ENhELzTg3RFb/2k++Lz/q6MlwqagzmE8kGZJ7fhbAK+V8apdJgiag0qM KWghCVaEkE1/FzwjWdmORcCIAUdXZ1UG4EOy5ytw9IAFsfnT8e6lJihPKeuNktAVj17B oZhvUrJd4BUZ5a2qgW3ZpIu0UsQQNMZCkRYO+J/idh6qOVwJaD2RQy9GgS12uR/LjmXE WrhSkDhzRm2m9Bj2A5yc47kCG5RhiSC6N69STnLvSbOGxXxRgt2zQ9A+kBXDTTiHpk63 S6FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3BHdWRw/I2mQJ5ddyQvsNmHD93DWER8WHM1dpXT9UjU=; b=QM0wub6qJQlDUAiisb980bEuBUfSuW41AM8JEeyBaNwUJc75/VxFZ3fpLg8skju9JY mjoXKLpTIrIljiZVhq7nDsQ7xkQWJIOkva9+nWwj+yzt1qwiRZvMtbROhwjGoBH+V4U2 3TrIgtUEZZTadDw5VtBjoWa9WJGNe0y28pxhhbrGz/qTrnlxFkP+fAtXuvA5my7JleGI U6fKGtDtf9Xzg/xS+a0+ZAfgeXI53RtCgx2OXBTqbSGhrOc4AKRqCnKAqBrqkpzTTLXP H+619G5qHiZdrf2CqXR9wlL4iPowhGJtqILxeSw+juwxG+K4dGnB8AW8O8CVqn7v5M3c Txng== X-Gm-Message-State: APjAAAV2CewNm7r/PamAxVQDASvvxrlf8gPVkFVBC88KI3NuwMghhz2i g8Vm02e+4LHeviXsMgbnHGVd5ZaE X-Google-Smtp-Source: APXvYqzkRz+yq085D5NwhxL0Ushkk2NYH9Kdf5zJCPAvFfNN6ayTkC2fDgyI6CdnJJcdqXXgaK22VQ== X-Received: by 2002:adf:ba94:: with SMTP id p20mr5879568wrg.213.1552746871205; Sat, 16 Mar 2019 07:34:31 -0700 (PDT) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id l8sm7518895wrv.45.2019.03.16.07.34.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 16 Mar 2019 07:34:30 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 16 Mar 2019 16:34:06 +0200 Message-Id: <20190316143421.8194-4-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190316143421.8194-1-marcel.apfelbaum@gmail.com> References: <20190316143421.8194-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 03/18] hw/rdma: Introduce protected qlist X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, kamalheib1@gmail.com, yuval.shaia@oracle.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia To make code more readable move handling of protected list to a rdma_utils Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Message-Id: <1552300155-25216-3-git-send-email-yuval.shaia@oracle.com> Reviewed-by: Kamal Heib Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 20 +++++-------------- hw/rdma/rdma_backend_defs.h | 8 ++------ hw/rdma/rdma_utils.c | 39 +++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_utils.h | 9 +++++++++ 4 files changed, 55 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d138591c86..37edf42215 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -527,9 +527,7 @@ static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, bctx->up_ctx = ctx; bctx->sge = *sge; - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + rdma_protected_qlist_append_int64(&backend_dev->recv_mads_list, bctx_id); return 0; } @@ -913,23 +911,19 @@ static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, static void process_incoming_mad_req(RdmaBackendDev *backend_dev, RdmaCmMuxMsg *msg) { - QObject *o_ctx_id; unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; trace_mad_message("recv", msg->umad.mad, msg->umad_len); - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - o_ctx_id = qlist_pop(backend_dev->recv_mads_list.list); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); - if (!o_ctx_id) { + cqe_ctx_id = rdma_protected_qlist_pop_int64(&backend_dev->recv_mads_list); + if (cqe_ctx_id == -ENOENT) { rdma_warn_report("No more free MADs buffers, waiting for a while"); sleep(THR_POLL_TO); return; } - cqe_ctx_id = qnum_get_uint(qobject_to(QNum, o_ctx_id)); bctx = rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); if (unlikely(!bctx)) { rdma_error_report("No matching ctx for req %ld", cqe_ctx_id); @@ -994,8 +988,7 @@ static int mad_init(RdmaBackendDev *backend_dev, CharBackend *mad_chr_be) return -EIO; } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); - backend_dev->recv_mads_list.list = qlist_new(); + rdma_protected_qlist_init(&backend_dev->recv_mads_list); enable_rdmacm_mux_async(backend_dev); @@ -1010,10 +1003,7 @@ static void mad_fini(RdmaBackendDev *backend_dev) { disable_rdmacm_mux_async(backend_dev); qemu_chr_fe_disconnect(backend_dev->rdmacm_mux.chr_be); - if (backend_dev->recv_mads_list.list) { - qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); - qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); - } + rdma_protected_qlist_destroy(&backend_dev->recv_mads_list); } int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 15ae8b970e..a8c15b09ab 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -20,6 +20,7 @@ #include "chardev/char-fe.h" #include #include "contrib/rdmacm-mux/rdmacm-mux.h" +#include "rdma_utils.h" typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -30,11 +31,6 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; -typedef struct RecvMadList { - QemuMutex lock; - QList *list; -} RecvMadList; - typedef struct RdmaCmMux { CharBackend *chr_be; int can_receive; @@ -48,7 +44,7 @@ typedef struct RdmaBackendDev { struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - RecvMadList recv_mads_list; + RdmaProtectedQList recv_mads_list; RdmaCmMux rdmacm_mux; } RdmaBackendDev; diff --git a/hw/rdma/rdma_utils.c b/hw/rdma/rdma_utils.c index b9f07fcda7..0a8abe572d 100644 --- a/hw/rdma/rdma_utils.c +++ b/hw/rdma/rdma_utils.c @@ -14,6 +14,8 @@ */ #include "qemu/osdep.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" #include "trace.h" #include "rdma_utils.h" @@ -51,3 +53,40 @@ void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len) pci_dma_unmap(dev, buffer, len, DMA_DIRECTION_TO_DEVICE, 0); } } + +void rdma_protected_qlist_init(RdmaProtectedQList *list) +{ + qemu_mutex_init(&list->lock); + list->list = qlist_new(); +} + +void rdma_protected_qlist_destroy(RdmaProtectedQList *list) +{ + if (list->list) { + qlist_destroy_obj(QOBJECT(list->list)); + qemu_mutex_destroy(&list->lock); + list->list = NULL; + } +} + +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value) +{ + qemu_mutex_lock(&list->lock); + qlist_append_int(list->list, value); + qemu_mutex_unlock(&list->lock); +} + +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list) +{ + QObject *obj; + + qemu_mutex_lock(&list->lock); + obj = qlist_pop(list->list); + qemu_mutex_unlock(&list->lock); + + if (!obj) { + return -ENOENT; + } + + return qnum_get_uint(qobject_to(QNum, obj)); +} diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index acd148837f..a8bf1d4fec 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -29,8 +29,17 @@ #define rdma_info_report(fmt, ...) \ info_report("%s: " fmt, "rdma", ## __VA_ARGS__) +typedef struct RdmaProtectedQList { + QemuMutex lock; + QList *list; +} RdmaProtectedQList; + void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); +void rdma_protected_qlist_init(RdmaProtectedQList *list); +void rdma_protected_qlist_destroy(RdmaProtectedQList *list); +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value); +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list); static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) {