From patchwork Fri Dec 15 16:48:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amrita H S X-Patchwork-Id: 1876691 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=LaB2Mfgb; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SsFXj4h1Dz23nF for ; Sat, 16 Dec 2023 03:48:37 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9F7053858D1E for ; Fri, 15 Dec 2023 16:48:35 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by sourceware.org (Postfix) with ESMTPS id 3E8B13858D1E for ; Fri, 15 Dec 2023 16:48:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3E8B13858D1E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: sourceware.org; spf=none smtp.mailfrom=linux.vnet.ibm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 3E8B13858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1702658907; cv=none; b=F3FijQ6WMkYr0M4TnGmPX2oAXC4s3rphpkWS+6yhV/792Jl4nXVCRyg0aLSUoGpIj7CmexifRJs4TcJDmScADsukL/gxMghITUNOA+/x+97j1sBgBXlXk5NCgNAFfxvItcNVbQTdbDlJfK52U+QHDWt8z1O8yrDRGnTRxZDNRTs= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1702658907; c=relaxed/simple; bh=zYgy4/0Lx3bKZyRb4TF+RB3PILKEwYCfGO0W81vYjyE=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=odj4dQ2jJysupFdLzydD407qvac0S4mxoC9mHiREDAGsPCAKuvZfkq3n+eYjV2/m0QdbYc6Jswlcd0Fv+A+Q772f8N1sFAUNf4ZUwIraV7r8ndkbpQYVF9EdLcGEC/9/Q+MGgKNDCFH1Z101OFFC7vYov2WvSpQAMcxyKJCIcBE= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFGgNxr013913 for ; Fri, 15 Dec 2023 16:48:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=4RIgv12fUYqTjH5YHltzUOWGEwsiARsL0XkMYjQU1uQ=; b=LaB2Mfgb4IPFYeiIAp///ASDrVwHK2hnmefmv3CFpcJtTcGuyPl8Pq2c958Bc0W6PJ/w ly/GpeJ6gE9xuboAghJNujpPIMmcoIJG78ur+iNIKE3pZcG5jRVX3HmhFKHPM5/BbHtE gN/fOg1NbWo82RVlCHnfOEY+uKerj5bzXD6vl2l4j8LEl5XjLLrhAxrb7Ke6Saq9kU5b IeCzMpLaT31nI7bhqmqj9Vh7aagkXfJO1jprNYmstiJ7KBsLhPK6bqya6mwSl4ETe9j5 t2eQkRGtZyNTXy/EDhkXGCgt1AAivd1p3+tFUmOeh/Ddvdk67PfkYuojX8/wtIwhQZrr QA== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3v0t20h94x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 15 Dec 2023 16:48:24 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3BFF96Ai005066 for ; Fri, 15 Dec 2023 16:48:23 GMT Received: from smtprelay01.fra02v.mail.ibm.com ([9.218.2.227]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3uw4sm108a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 15 Dec 2023 16:48:23 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay01.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3BFGmLen10748524 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 15 Dec 2023 16:48:21 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0A8702004B; Fri, 15 Dec 2023 16:48:21 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4BC7420040; Fri, 15 Dec 2023 16:48:20 +0000 (GMT) Received: from ltcd97-lp3.. (unknown [9.40.194.171]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 15 Dec 2023 16:48:20 +0000 (GMT) From: Amrita H S To: libc-alpha@sourceware.org Cc: Amrita H S , "Paul E . Murphy" Subject: [PATCH V4] powerpc: Fix performance issues of strcmp power10 Date: Fri, 15 Dec 2023 11:48:17 -0500 Message-ID: <20231215164817.1982535-1-amritahs@linux.vnet.ibm.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: cAOScde_CeXNHK1rtNaFDocgfK8aldGU X-Proofpoint-ORIG-GUID: cAOScde_CeXNHK1rtNaFDocgfK8aldGU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-15_10,2023-12-14_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 malwarescore=0 clxscore=1015 lowpriorityscore=0 phishscore=0 adultscore=0 spamscore=0 priorityscore=1501 impostorscore=0 bulkscore=0 suspectscore=0 mlxlogscore=721 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312150116 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, KAM_NUMSUBJECT, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Current implementation of strcmp for power10 has performance regression for multiple small sizes and alignment combination. Most of these performance issues are fixed by this patch. The compare loop is unrolled and page crosses of unrolled loop is handled. Thanks to Paul E. Murphy for helping in fixing the performance issues. Signed-off-by: Amrita H S Co-Authored-By: Paul E. Murphy Reviewed-by: Rajalakshmi Srinivasaraghavan --- sysdeps/powerpc/powerpc64/le/power10/strcmp.S | 161 +++++++++++------- 1 file changed, 95 insertions(+), 66 deletions(-) diff --git a/sysdeps/powerpc/powerpc64/le/power10/strcmp.S b/sysdeps/powerpc/powerpc64/le/power10/strcmp.S index a3c1adad53..3406f4f26a 100644 --- a/sysdeps/powerpc/powerpc64/le/power10/strcmp.S +++ b/sysdeps/powerpc/powerpc64/le/power10/strcmp.S @@ -62,7 +62,7 @@ lxvl 32+v5,reg2,r0; \ add reg1,reg1,len_reg; \ add reg2,reg2,len_reg; \ - vcmpnezb. v7,v4,v5; \ + vcmpnezb v7,v4,v5; \ vctzlsbb r6,v7; \ cmpld cr7,r6,len_reg; \ blt cr7,L(different); \ @@ -72,70 +72,110 @@ .machine power9 ENTRY_TOCLESS (STRCMP, 4) - li r11,16 - /* eq bit of cr1 used as swap status flag to indicate if - source pointers were swapped. */ - crclr 4*cr1+eq - vspltisb v19,-1 - andi. r7,r3,15 - sub r7,r11,r7 /* r7(nalign1) = 16 - (str1 & 15). */ - andi. r9,r4,15 - sub r5,r11,r9 /* r5(nalign2) = 16 - (str2 & 15). */ - cmpld cr7,r7,r5 - beq cr7,L(same_aligned) - blt cr7,L(nalign1_min) - /* Swap r3 and r4, and r7 and r5 such that r3 and r7 hold the - pointer which is closer to the next 16B boundary so that only - one CHECK_N_BYTES is needed before entering the loop below. */ - mr r8,r4 - mr r4,r3 - mr r3,r8 - mr r12,r7 - mr r7,r5 - mr r5,r12 - crset 4*cr1+eq /* Set bit on swapping source pointers. */ + andi. r7,r3,4095 + andi. r8,r4,4095 + cmpldi cr0,r7,4096-16 + cmpldi cr1,r8,4096-16 + bgt cr0,L(crosses) + bgt cr1,L(crosses) + COMPARE_16(v4,v5,0) - .p2align 5 +L(crosses): + andi. r7,r3,15 + subfic r7,r7,16 /* r7(nalign1) = 16 - (str1 & 15). */ + andi. r9,r4,15 + subfic r5,r9,16 /* r5(nalign2) = 16 - (str2 & 15). */ + cmpld cr7,r7,r5 + beq cr7,L(same_aligned) + blt cr7,L(nalign1_min) + + /* nalign2 is minimum and s2 pointer is aligned. */ + CHECK_N_BYTES(r3,r4,r5) + /* Are we on the 64B hunk which crosses a page? */ + andi. r10,r3,63 /* Determine offset into 64B hunk. */ + andi. r8,r3,15 /* The offset into the 16B hunk. */ + neg r7,r3 + andi. r9,r7,15 /* Number of bytes after a 16B cross. */ + rlwinm. r7,r7,26,0x3F /* ((r3-4096))>>6&63. */ + beq L(compare_64_pagecross) + mtctr r7 + b L(compare_64B_unaligned) + + /* nalign1 is minimum and s1 pointer is aligned. */ L(nalign1_min): CHECK_N_BYTES(r3,r4,r7) + /* Are we on the 64B hunk which crosses a page? */ + andi. r10,r4,63 /* Determine offset into 64B hunk. */ + andi. r8,r4,15 /* The offset into the 16B hunk. */ + neg r7,r4 + andi. r9,r7,15 /* Number of bytes after a 16B cross. */ + rlwinm. r7,r7,26,0x3F /* ((r4-4096))>>6&63. */ + beq L(compare_64_pagecross) + mtctr r7 .p2align 5 -L(s1_aligned): - /* r9 and r5 is number of bytes to be read after and before - page boundary correspondingly. */ - sub r5,r5,r7 - subfic r9,r5,16 - /* Now let r7 hold the count of quadwords which can be - checked without crossing a page boundary. quadword offset is - (str2>>4)&0xFF. */ - rlwinm r7,r4,28,0xFF - /* Below check is required only for first iteration. For second - iteration and beyond, the new loop counter is always 255. */ - cmpldi r7,255 - beq L(L3) - /* Get the initial loop count by 255-((str2>>4)&0xFF). */ - subfic r11,r7,255 +L(compare_64B_unaligned): + COMPARE_16(v4,v5,0) + COMPARE_16(v4,v5,16) + COMPARE_16(v4,v5,32) + COMPARE_16(v4,v5,48) + addi r3,r3,64 + addi r4,r4,64 + bdnz L(compare_64B_unaligned) - .p2align 5 -L(L1): + /* Cross the page boundary of s2, carefully. Only for first + iteration we have to get the count of 64B blocks to be checked. + From second iteration and beyond, loop counter is always 63. */ +L(compare_64_pagecross): + li r11, 63 mtctr r11 - - .p2align 5 -L(L2): - COMPARE_16(v4,v5,0) /* Load 16B blocks using lxv. */ + cmpldi r10,16 + ble L(cross_4) + cmpldi r10,32 + ble L(cross_3) + cmpldi r10,48 + ble L(cross_2) +L(cross_1): + CHECK_N_BYTES(r3,r4,r9) + CHECK_N_BYTES(r3,r4,r8) + COMPARE_16(v4,v5,0) + COMPARE_16(v4,v5,16) + COMPARE_16(v4,v5,32) + addi r3,r3,48 + addi r4,r4,48 + b L(compare_64B_unaligned) +L(cross_2): + COMPARE_16(v4,v5,0) addi r3,r3,16 addi r4,r4,16 - bdnz L(L2) - /* Cross the page boundary of s2, carefully. */ - - .p2align 5 -L(L3): - CHECK_N_BYTES(r3,r4,r5) CHECK_N_BYTES(r3,r4,r9) - li r11,255 /* Load the new loop counter. */ - b L(L1) + CHECK_N_BYTES(r3,r4,r8) + COMPARE_16(v4,v5,0) + COMPARE_16(v4,v5,16) + addi r3,r3,32 + addi r4,r4,32 + b L(compare_64B_unaligned) +L(cross_3): + COMPARE_16(v4,v5,0) + COMPARE_16(v4,v5,16) + addi r3,r3,32 + addi r4,r4,32 + CHECK_N_BYTES(r3,r4,r9) + CHECK_N_BYTES(r3,r4,r8) + COMPARE_16(v4,v5,0) + addi r3,r3,16 + addi r4,r4,16 + b L(compare_64B_unaligned) +L(cross_4): + COMPARE_16(v4,v5,0) + COMPARE_16(v4,v5,16) + COMPARE_16(v4,v5,32) + addi r3,r3,48 + addi r4,r4,48 + CHECK_N_BYTES(r3,r4,r9) + CHECK_N_BYTES(r3,r4,r8) + b L(compare_64B_unaligned) - .p2align 5 L(same_aligned): CHECK_N_BYTES(r3,r4,r7) /* Align s1 to 32B and adjust s2 address. @@ -168,18 +208,7 @@ L(16B_aligned_loop): /* Calculate and return the difference. */ L(different): - vctzlsbb r6,v7 - vextubrx r5,r6,v4 - vextubrx r4,r6,v5 - bt 4*cr1+eq,L(swapped) - subf r3,r4,r5 - blr - - /* If src pointers were swapped, then swap the - indices and calculate the return value. */ -L(swapped): - subf r3,r5,r4 - blr + TAIL(v4,v5) .p2align 5 L(32B_aligned_loop):