From patchwork Tue Jun 18 20:35:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Pengxuan Zheng X-Patchwork-Id: 1949464 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=quicinc.com header.i=@quicinc.com header.a=rsa-sha256 header.s=qcppdkim1 header.b=mrK+9ymM; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W3dp44fclz20KL for ; Wed, 19 Jun 2024 06:36:44 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C8B4D388302B for ; Tue, 18 Jun 2024 20:36:42 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by sourceware.org (Postfix) with ESMTPS id 258283883030 for ; Tue, 18 Jun 2024 20:36:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 258283883030 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=quicinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=quicinc.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 258283883030 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=205.220.180.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718742980; cv=none; b=c785xHcQ46QukLOCrsBI9jBamM2l3aCdMeKPs9Ag31uoKhuLiJ6Rp5ygJLVfZErSk2rUWSnV1tS6JnWa4ipV+ITTnNAxKSc0v/SkPwmP0niSTg1yQvIE6VwOcPRGY1JcbIpg19KUAzXr/jzFoatmEyWpyLnpHWCSFxsAWmb1KPU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718742980; c=relaxed/simple; bh=ZWBA/D+9maM96Re6jANh61Rd54blAFMlNZC0b++rgdc=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=apl1fTa8V+EbKjY4OjHk9DohNh3WPreE05f0s903fl4rCtedwenQxHec91mGlg9svPs/sD9DUupRahs9OJIJRwTcWPRXesKpP/JxcZUjKQjrkBJI+l3e+tr5P3G72Edo68yBZ9R2E56Y4Rc/dknEI26m9t8xywucmG1PeH05RIw= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from pps.filterd (m0279869.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45IBDIbg006973 for ; Tue, 18 Jun 2024 20:36:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h= cc:content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to; s=qcppdkim1; bh=NYU2RP+3s7UNorR7Y8OLL2 kp6L0LSjlGJB2xNEu7tTg=; b=mrK+9ymMhGc66Y+NaDEUFGdHcInw3+BGuCrFZh xowRfo1bhSQh3ZQLWtxorAJchcQmKWeASdJnhE+dmhqznV4orjeicOWZvGRISkvR iLENjwHjZViiP5BA5WCpmlL1A+oPUqWxLOUQk+Xl0zqSPdp/dc9nGfkV/AFiH+BD EMn9N3QDkyA7VnDq70JpA1W4tLtCzSEnDxxsJMJuDyK1NwEYpza/6nabqwmgVJW5 r5vWV9glmo2Gb0QdwUyDnoRVXgDVxlLEHUjOBNqwGiV7mLdUWEB99MXswd1pTEeA X4SuKruLe0YkkdNtBJ9cJCA5/QaDN8ZzT2Ek2AjyFO0RArzQ== Received: from nalasppmta04.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3yu95rhcsa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 18 Jun 2024 20:36:17 +0000 (GMT) Received: from nalasex01c.na.qualcomm.com (nalasex01c.na.qualcomm.com [10.47.97.35]) by NALASPPMTA04.qualcomm.com (8.17.1.19/8.17.1.19) with ESMTPS id 45IKaGu6002068 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 18 Jun 2024 20:36:16 GMT Received: from hu-pzheng-lv.qualcomm.com (10.49.16.6) by nalasex01c.na.qualcomm.com (10.47.97.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.9; Tue, 18 Jun 2024 13:36:15 -0700 From: Pengxuan Zheng To: CC: Pengxuan Zheng Subject: [PATCH v5] aarch64: Add vector popcount besides QImode [PR113859] Date: Tue, 18 Jun 2024 13:35:54 -0700 Message-ID: <20240618203554.20129-1-quic_pzheng@quicinc.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01b.na.qualcomm.com (10.47.209.197) To nalasex01c.na.qualcomm.com (10.47.97.35) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: nFiIBoWFgzXyaKEwHQGJV95V8DCIFzbe X-Proofpoint-ORIG-GUID: nFiIBoWFgzXyaKEwHQGJV95V8DCIFzbe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-18_03,2024-06-17_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 bulkscore=0 mlxscore=0 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 phishscore=0 mlxlogscore=762 suspectscore=0 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2405170001 definitions=main-2406180151 X-Spam-Status: No, score=-13.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This patch improves GCC’s vectorization of __builtin_popcount for aarch64 target by adding popcount patterns for vector modes besides QImode, i.e., HImode, SImode and DImode. With this patch, we now generate the following for V8HI: cnt v1.16b, v0.16b uaddlp v2.8h, v1.16b For V4HI, we generate: cnt v1.8b, v0.8b uaddlp v2.4h, v1.8b For V4SI, we generate: cnt v1.16b, v0.16b uaddlp v2.8h, v1.16b uaddlp v3.4s, v2.8h For V4SI with TARGET_DOTPROD, we generate the following instead: movi v0.4s, #0 movi v1.16b, #1 cnt v3.16b, v2.16b udot v0.4s, v3.16b, v1.16b For V2SI, we generate: cnt v1.8b, v.8b uaddlp v2.4h, v1.8b uaddlp v3.2s, v2.4h For V2SI with TARGET_DOTPROD, we generate the following instead: movi v0.8b, #0 movi v1.8b, #1 cnt v3.8b, v2.8b udot v0.2s, v3.8b, v1.8b For V2DI, we generate: cnt v1.16b, v.16b uaddlp v2.8h, v1.16b uaddlp v3.4s, v2.8h uaddlp v4.2d, v3.4s For V4SI with TARGET_DOTPROD, we generate the following instead: movi v0.4s, #0 movi v1.16b, #1 cnt v3.16b, v2.16b udot v0.4s, v3.16b, v1.16b uaddlp v0.2d, v0.4s PR target/113859 gcc/ChangeLog: * config/aarch64/aarch64-simd.md (aarch64_addlp): Rename to... (@aarch64_addlp): ... This. (popcount2): New define_expand. gcc/testsuite/ChangeLog: * gcc.target/aarch64/popcnt-udot.c: New test. * gcc.target/aarch64/popcnt-vec.c: New test. Signed-off-by: Pengxuan Zheng --- gcc/config/aarch64/aarch64-simd.md | 51 +++++++++++++- .../gcc.target/aarch64/popcnt-udot.c | 58 ++++++++++++++++ gcc/testsuite/gcc.target/aarch64/popcnt-vec.c | 69 +++++++++++++++++++ 3 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/popcnt-udot.c create mode 100644 gcc/testsuite/gcc.target/aarch64/popcnt-vec.c diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md index 0bb39091a38..1c76123a518 100644 --- a/gcc/config/aarch64/aarch64-simd.md +++ b/gcc/config/aarch64/aarch64-simd.md @@ -3461,7 +3461,7 @@ (define_insn "*aarch64_addlv_ze" [(set_attr "type" "neon_reduc_add")] ) -(define_expand "aarch64_addlp" +(define_expand "@aarch64_addlp" [(set (match_operand: 0 "register_operand") (plus: (vec_select: @@ -3517,6 +3517,55 @@ (define_insn "popcount2" [(set_attr "type" "neon_cnt")] ) +(define_expand "popcount2" + [(set (match_operand:VDQHSD 0 "register_operand") + (popcount:VDQHSD (match_operand:VDQHSD 1 "register_operand")))] + "TARGET_SIMD" + { + /* Generate a byte popcount. */ + machine_mode mode = == 64 ? V8QImode : V16QImode; + rtx tmp = gen_reg_rtx (mode); + auto icode = optab_handler (popcount_optab, mode); + emit_insn (GEN_FCN (icode) (tmp, gen_lowpart (mode, operands[1]))); + + if (TARGET_DOTPROD) + { + /* For V4SI and V2SI, we can generate a UDOT with a 0 accumulator and a + 1 multiplicand. For V2DI, another UAADDLP is needed. */ + if (mode == SImode || mode == DImode) + { + machine_mode dp_mode = == 64 ? V2SImode : V4SImode; + rtx ones = force_reg (mode, CONST1_RTX (mode)); + rtx zeros = CONST0_RTX (dp_mode); + rtx dp = gen_reg_rtx (dp_mode); + auto dp_icode = optab_handler (udot_prod_optab, mode); + emit_move_insn (dp, zeros); + emit_insn (GEN_FCN (dp_icode) (dp, tmp, ones, dp)); + if (mode == V2DImode) + { + emit_insn (gen_aarch64_uaddlpv4si (operands[0], dp)); + DONE; + } + emit_move_insn (operands[0], dp); + DONE; + } + } + + /* Use a sequence of UADDLPs to accumulate the counts. Each step doubles + the element size and halves the number of elements. */ + do + { + auto icode = code_for_aarch64_addlp (ZERO_EXTEND, GET_MODE (tmp)); + mode = insn_data[icode].operand[0].mode; + rtx dest = mode == mode ? operands[0] : gen_reg_rtx (mode); + emit_insn (GEN_FCN (icode) (dest, tmp)); + tmp = dest; + } + while (mode != mode); + DONE; + } +) + ;; 'across lanes' max and min ops. ;; Template for outputting a scalar, so we can create __builtins which can be diff --git a/gcc/testsuite/gcc.target/aarch64/popcnt-udot.c b/gcc/testsuite/gcc.target/aarch64/popcnt-udot.c new file mode 100644 index 00000000000..150ff746361 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/popcnt-udot.c @@ -0,0 +1,58 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -march=armv8.2-a+dotprod -fno-vect-cost-model" } */ + +/* +** bar: +** ldr q([0-9]+), \[x0\] +** movi v([0-9]+).16b, 0x1 +** movi v([0-9]+).4s, 0 +** cnt v([0-9]+).16b, v\1.16b +** udot v\3.4s, v\4.16b, v\2.16b +** str q\3, \[x1\] +** ret +*/ +void +bar (unsigned int *__restrict b, unsigned int *__restrict d) +{ + d[0] = __builtin_popcount (b[0]); + d[1] = __builtin_popcount (b[1]); + d[2] = __builtin_popcount (b[2]); + d[3] = __builtin_popcount (b[3]); +} + +/* +** bar1: +** ldr d([0-9]+), \[x0\] +** movi v([0-9]+).8b, 0x1 +** movi v([0-9]+).2s, 0 +** cnt v([0-9]+).8b, v\1.8b +** udot v\3.2s, v\4.8b, v\2.8b +** str d\3, \[x1\] +** ret +*/ +void +bar1 (unsigned int *__restrict b, unsigned int *__restrict d) +{ + d[0] = __builtin_popcount (b[0]); + d[1] = __builtin_popcount (b[1]); +} + +/* +** bar2: +** ldr q([0-9]+), \[x0\] +** movi v([0-9]+).16b, 0x1 +** movi v([0-9]+).4s, 0 +** cnt v([0-9]+).16b, v\1.16b +** udot v\3.4s, v\4.16b, v\2.16b +** uaddlp v\3.2d, v\3.4s +** str q\3, \[x1\] +** ret +*/ +void +bar2 (unsigned long long *__restrict b, unsigned long long *__restrict d) +{ + d[0] = __builtin_popcountll (b[0]); + d[1] = __builtin_popcountll (b[1]); +} + +/* { dg-final { check-function-bodies "**" "" "" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/popcnt-vec.c b/gcc/testsuite/gcc.target/aarch64/popcnt-vec.c new file mode 100644 index 00000000000..0c4926d7ca8 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/popcnt-vec.c @@ -0,0 +1,69 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fno-vect-cost-model" } */ + +/* This function should produce cnt v.16b. */ +void +bar (unsigned char *__restrict b, unsigned char *__restrict d) +{ + for (int i = 0; i < 1024; i++) + d[i] = __builtin_popcount (b[i]); +} + +/* This function should produce cnt v.16b and uaddlp (Add Long Pairwise). */ +void +bar1 (unsigned short *__restrict b, unsigned short *__restrict d) +{ + for (int i = 0; i < 1024; i++) + d[i] = __builtin_popcount (b[i]); +} + +/* This function should produce cnt v.16b and 2 uaddlp (Add Long Pairwise). */ +void +bar2 (unsigned int *__restrict b, unsigned int *__restrict d) +{ + for (int i = 0; i < 1024; i++) + d[i] = __builtin_popcount (b[i]); +} + +/* This function should produce cnt v.16b and 3 uaddlp (Add Long Pairwise). */ +void +bar3 (unsigned long long *__restrict b, unsigned long long *__restrict d) +{ + for (int i = 0; i < 1024; i++) + d[i] = __builtin_popcountll (b[i]); +} + +/* SLP + This function should produce cnt v.8b and uaddlp (Add Long Pairwise). */ +void +bar4 (unsigned short *__restrict b, unsigned short *__restrict d) +{ + d[0] = __builtin_popcount (b[0]); + d[1] = __builtin_popcount (b[1]); + d[2] = __builtin_popcount (b[2]); + d[3] = __builtin_popcount (b[3]); +} + +/* SLP + This function should produce cnt v.8b and 2 uaddlp (Add Long Pairwise). */ +void +bar5 (unsigned int *__restrict b, unsigned int *__restrict d) +{ + d[0] = __builtin_popcount (b[0]); + d[1] = __builtin_popcount (b[1]); +} + +/* SLP + This function should produce cnt v.16b and 3 uaddlp (Add Long Pairwise). */ +void +bar6 (unsigned long long *__restrict b, unsigned long long *__restrict d) +{ + d[0] = __builtin_popcountll (b[0]); + d[1] = __builtin_popcountll (b[1]); +} + +/* { dg-final { scan-assembler-not {\tbl\tpopcount} } } */ +/* { dg-final { scan-assembler-times {cnt\t} 7 } } */ +/* { dg-final { scan-assembler-times {uaddlp\t} 12 } } */ +/* { dg-final { scan-assembler-times {ldr\tq} 5 } } */ +/* { dg-final { scan-assembler-times {ldr\td} 2 } } */