From patchwork Mon Jun 17 17:31:03 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kamal Mostafa X-Patchwork-Id: 252004 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) by ozlabs.org (Postfix) with ESMTP id 0195A2C00A0 for ; Tue, 18 Jun 2013 03:37:16 +1000 (EST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1UodMc-0004jA-0f; Mon, 17 Jun 2013 17:37:10 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1UodLL-0003ve-77 for kernel-team@lists.ubuntu.com; Mon, 17 Jun 2013 17:35:51 +0000 Received: from c-67-160-231-42.hsd1.ca.comcast.net ([67.160.231.42] helo=fourier) by youngberry.canonical.com with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1UodHL-0007F4-BR; Mon, 17 Jun 2013 17:31:43 +0000 Received: from kamal by fourier with local (Exim 4.80) (envelope-from ) id 1UodHJ-00068C-C6; Mon, 17 Jun 2013 10:31:41 -0700 From: Kamal Mostafa To: linux-kernel@vger.kernel.org, stable@vger.kernel.org, kernel-team@lists.ubuntu.com Subject: [PATCH 59/84] libceph: must hold mutex for reset_changed_osds() Date: Mon, 17 Jun 2013 10:31:03 -0700 Message-Id: <1371490288-23167-60-git-send-email-kamal@canonical.com> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1371490288-23167-1-git-send-email-kamal@canonical.com> References: <1371490288-23167-1-git-send-email-kamal@canonical.com> X-Extended-Stable: 3.8 Cc: Alex Elder , Kamal Mostafa X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: kernel-team-bounces@lists.ubuntu.com 3.8.13.3 -stable review patch. If anyone has any objections, please let me know. ------------------ From: Alex Elder commit 14d2f38df67fadee34625fcbd282ee22514c4846 upstream. An osd client has a red-black tree describing its osds, and occasionally we would get crashes due to one of these trees tree becoming corrupt somehow. The problem turned out to be that reset_changed_osds() was being called without protection of the osd client request mutex. That function would call __reset_osd() for any osd that had changed, and __reset_osd() would call __remove_osd() for any osd with no outstanding requests, and finally __remove_osd() would remove the corresponding entry from the red-black tree. Thus, the tree was getting modified without having any lock protection, and was vulnerable to problems due to concurrent updates. This appears to be the only osd tree updating path that has this problem. It can be fairly easily fixed by moving the call up a few lines, to just before the request mutex gets dropped in kick_requests(). This resolves: http://tracker.ceph.com/issues/5043 Signed-off-by: Alex Elder Reviewed-by: Sage Weil Signed-off-by: Kamal Mostafa --- net/ceph/osd_client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index eb9a444..b7b980d 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -1339,13 +1339,13 @@ static void kick_requests(struct ceph_osd_client *osdc, int force_resend) __register_request(osdc, req); __unregister_linger_request(osdc, req); } + reset_changed_osds(osdc); mutex_unlock(&osdc->request_mutex); if (needmap) { dout("%d requests for down osds, need new map\n", needmap); ceph_monc_request_next_osdmap(&osdc->client->monc); } - reset_changed_osds(osdc); }