Message ID | 20230926110009.462218-1-pan2.li@intel.com |
---|---|
State | New |
Headers | show |
Series | [v1] RISC-V: Support FP round auto-vectorization | expand |
LGTM juzhe.zhong@rivai.ai From: pan2.li Date: 2023-09-26 19:00 To: gcc-patches CC: juzhe.zhong; pan2.li; yanzhang.wang; kito.cheng Subject: [PATCH v1] RISC-V: Support FP round auto-vectorization From: Pan Li <pan2.li@intel.com> This patch would like to support auto-vectorization for the round API in math.h. It depends on the -ffast-math option. When we would like to call round/roundf like v2 = round (v1), we will convert it into below insns (reference the implementation of llvm). * vfcvt.x.f v3, v1, RMM * vfcvt.f.x v2, v3 However, the floating point value may not need the cvt as above if its mantissa is zero. Take single precision floating point as example: +------------+---------------+-----------------+ | raw float | binary layout | after round | +------------+---------------+-----------------+ | -8388607.5 | 0xcaffffff | -8388608.0 | | 8388607.5 | 0x4affffff | 8388608.0 | | 8388608.0 | 0x4b000000 | 8388608.0 | | 8388609.0 | 0x4b000001 | 8388609.0 | +------------+---------------+-----------------+ All single floating point >= 8388608.0 will have all zero mantisaa. We leverage vmflt and mask to filter them out in vector and only do the cvt on mask. Befor this patch: math-round-1.c:21:1: missed: couldn't vectorize loop ... .L3: flw fa0,0(s0) addi s0,s0,4 addi s1,s1,4 call round fsw fa0,-4(s1) bne s0,s2,.L3 After this patch: ... fsrmi 4 // RMM, rounding to nearest, ties to max magnitude .L4: vfabs.v v2,v1 vmflt.vf v0,v2,fa5 vfcvt.x.f.v v4,v1,v0.t vfcvt.f.x.v v2,v4,v0.t vfsgnj.vv v2,v2,v1 bne .L4 .L14: fsrm a6 ret Please note VLS mode is also involved in this patch and covered by the test cases. gcc/ChangeLog: * config/riscv/autovec.md (round<mode>2): New pattern. * config/riscv/riscv-protos.h (enum insn_flags): New enum type. (enum insn_type): Ditto. (expand_vec_round): New function decl. * config/riscv/riscv-v.cc (expand_vec_round): New function impl. gcc/testsuite/ChangeLog: * gcc.target/riscv/rvv/autovec/unop/math-round-0.c: New test. * gcc.target/riscv/rvv/autovec/unop/math-round-1.c: New test. * gcc.target/riscv/rvv/autovec/unop/math-round-2.c: New test. * gcc.target/riscv/rvv/autovec/unop/math-round-3.c: New test. * gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c: New test. * gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c: New test. * gcc.target/riscv/rvv/autovec/vls/math-round-1.c: New test. Signed-off-by: Pan Li <pan2.li@intel.com> --- gcc/config/riscv/autovec.md | 10 ++++ gcc/config/riscv/riscv-protos.h | 5 ++ gcc/config/riscv/riscv-v.cc | 24 ++++++++ .../riscv/rvv/autovec/unop/math-round-0.c | 23 ++++++++ .../riscv/rvv/autovec/unop/math-round-1.c | 23 ++++++++ .../riscv/rvv/autovec/unop/math-round-2.c | 23 ++++++++ .../riscv/rvv/autovec/unop/math-round-3.c | 25 +++++++++ .../riscv/rvv/autovec/unop/math-round-run-1.c | 39 +++++++++++++ .../riscv/rvv/autovec/unop/math-round-run-2.c | 39 +++++++++++++ .../riscv/rvv/autovec/vls/math-round-1.c | 56 +++++++++++++++++++ 10 files changed, 267 insertions(+) create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index 1d2fca60e98..798cf1272c5 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -2251,3 +2251,13 @@ (define_expand "rint<mode>2" DONE; } ) + +(define_expand "round<mode>2" + [(match_operand:V_VLSF 0 "register_operand") + (match_operand:V_VLSF 1 "register_operand")] + "TARGET_VECTOR && !flag_trapping_math && !flag_rounding_math" + { + riscv_vector::expand_vec_round (operands[0], operands[1], <MODE>mode, <VCONVERT>mode); + DONE; + } +) diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h index 629adeea94c..70ca244c591 100644 --- a/gcc/config/riscv/riscv-protos.h +++ b/gcc/config/riscv/riscv-protos.h @@ -256,6 +256,9 @@ enum insn_flags : unsigned int /* Means INSN has FRM operand and the value is FRM_RDN. */ FRM_RDN_P = 1 << 17, + + /* Means INSN has FRM operand and the value is FRM_RMM. */ + FRM_RMM_P = 1 << 18, }; enum insn_type : unsigned int @@ -299,6 +302,7 @@ enum insn_type : unsigned int UNARY_OP_TAMU_FRM_DYN = UNARY_OP_TAMU | FRM_DYN_P, UNARY_OP_TAMU_FRM_RUP = UNARY_OP_TAMU | FRM_RUP_P, UNARY_OP_TAMU_FRM_RDN = UNARY_OP_TAMU | FRM_RDN_P, + UNARY_OP_TAMU_FRM_RMM = UNARY_OP_TAMU | FRM_RMM_P, /* Binary operator. */ BINARY_OP = __NORMAL_OP | BINARY_OP_P, @@ -463,6 +467,7 @@ void expand_vec_ceil (rtx, rtx, machine_mode, machine_mode); void expand_vec_floor (rtx, rtx, machine_mode, machine_mode); void expand_vec_nearbyint (rtx, rtx, machine_mode, machine_mode); void expand_vec_rint (rtx, rtx, machine_mode, machine_mode); +void expand_vec_round (rtx, rtx, machine_mode, machine_mode); #endif bool sew64_scalar_helper (rtx *, rtx *, rtx, machine_mode, bool, void (*)(rtx *, rtx)); diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc index 445ed000f88..5f738634219 100644 --- a/gcc/config/riscv/riscv-v.cc +++ b/gcc/config/riscv/riscv-v.cc @@ -330,6 +330,8 @@ public: add_rounding_mode_operand (FRM_RUP); else if (m_insn_flags & FRM_RDN_P) add_rounding_mode_operand (FRM_RDN); + else if (m_insn_flags & FRM_RMM_P) + add_rounding_mode_operand (FRM_RMM); gcc_assert (insn_data[(int) icode].n_operands == m_opno); expand (icode, any_mem_p); @@ -3720,4 +3722,26 @@ expand_vec_rint (rtx op_0, rtx op_1, machine_mode vec_fp_mode, emit_vec_copysign (op_0, op_0, op_1, vec_fp_mode); } +void +expand_vec_round (rtx op_0, rtx op_1, machine_mode vec_fp_mode, + machine_mode vec_int_mode) +{ + /* Step-1: Get the abs float value for mask generation. */ + emit_vec_abs (op_0, op_1, vec_fp_mode); + + /* Step-2: Generate the mask on const fp. */ + rtx const_fp = get_fp_rounding_coefficient (GET_MODE_INNER (vec_fp_mode)); + rtx mask = emit_vec_float_cmp_mask (op_0, LT, const_fp, vec_fp_mode); + + /* Step-3: Convert to integer on mask, rounding to nearest (aka round). */ + rtx tmp = gen_reg_rtx (vec_int_mode); + emit_vec_cvt_x_f (tmp, op_1, mask, UNARY_OP_TAMU_FRM_RMM, vec_fp_mode); + + /* Step-4: Convert to floating-point on mask for the round result. */ + emit_vec_cvt_f_x (op_0, tmp, mask, UNARY_OP_TAMU_FRM_RMM, vec_fp_mode); + + /* Step-5: Retrieve the sign bit for -0.0. */ + emit_vec_copysign (op_0, op_0, op_1, vec_fp_mode); +} + } // namespace riscv_vector diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c new file mode 100644 index 00000000000..06de57bf7e2 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv_zvfh -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test__Float16___builtin_roundf16: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e16,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (_Float16, __builtin_roundf16) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c new file mode 100644 index 00000000000..ee51bcd820b --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_float___builtin_roundf: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e32,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (float, __builtin_roundf) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c new file mode 100644 index 00000000000..d78f0583e41 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_double___builtin_round: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e64,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (double, __builtin_round) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c new file mode 100644 index 00000000000..98d14673e20 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_float___builtin_roundf: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e32,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** vmerge\.vvm\s+v[0-9]+,\s*v[0-9]+,\s*v[0-9]+,\s*v0 +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_COND_UNARY_CALL (float, __builtin_roundf) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c new file mode 100644 index 00000000000..baaa614879a --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c @@ -0,0 +1,39 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "-std=c99 -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math" } */ + +#include "test-math.h" + +#define ARRAY_SIZE 128 + +float in[ARRAY_SIZE]; +float out[ARRAY_SIZE]; +float ref[ARRAY_SIZE]; + +TEST_UNARY_CALL (float, __builtin_roundf) +TEST_ASSERT (float) + +TEST_INIT (float, 1.2, 1.0, 1) +TEST_INIT (float, -1.6, -2.0, 2) +TEST_INIT (float, 3.0, 3.0, 3) +TEST_INIT (float, 8388607.5, 8388608.0, 4) +TEST_INIT (float, 8388609.0, 8388609.0, 5) +TEST_INIT (float, 0.0, 0.0, 6) +TEST_INIT (float, -0.0, -0.0, 7) +TEST_INIT (float, -8388607.5, -8388608.0, 8) +TEST_INIT (float, -8388608.0, -8388608.0, 9) + +int +main () +{ + RUN_TEST (float, 1, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 2, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 3, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 4, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 5, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 6, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 7, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 8, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 9, __builtin_roundf, in, out, ref, ARRAY_SIZE); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c new file mode 100644 index 00000000000..8ed82525814 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c @@ -0,0 +1,39 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "-std=c99 -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math" } */ + +#include "test-math.h" + +#define ARRAY_SIZE 128 + +double in[ARRAY_SIZE]; +double out[ARRAY_SIZE]; +double ref[ARRAY_SIZE]; + +TEST_UNARY_CALL (double, __builtin_round) +TEST_ASSERT (double) + +TEST_INIT (double, 1.2, 1.0, 1) +TEST_INIT (double, -1.8, -2.0, 2) +TEST_INIT (double, 3.0, 3.0, 3) +TEST_INIT (double, 4503599627370495.5, 4503599627370496.0, 4) +TEST_INIT (double, 4503599627370497.0, 4503599627370497.0, 5) +TEST_INIT (double, 0.0, 0.0, 6) +TEST_INIT (double, -0.0, -0.0, 7) +TEST_INIT (double, -4503599627370495.5, -4503599627370496.0, 8) +TEST_INIT (double, -4503599627370496.0, -4503599627370496.0, 9) + +int +main () +{ + RUN_TEST (double, 1, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 2, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 3, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 4, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 5, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 6, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 7, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 8, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 9, __builtin_round, in, out, ref, ARRAY_SIZE); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c new file mode 100644 index 00000000000..97fd6975969 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c @@ -0,0 +1,56 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv_zvfh_zvl4096b -mabi=lp64d -O3 --param=riscv-autovec-lmul=m8 -ffast-math -fdump-tree-optimized" } */ + +#include "def.h" + +DEF_OP_V (roundf16, 1, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 2, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 4, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 8, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 16, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 32, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 64, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 128, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 256, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 512, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 1024, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 2048, _Float16, __builtin_roundf16) + +DEF_OP_V (roundf, 1, float, __builtin_roundf) +DEF_OP_V (roundf, 2, float, __builtin_roundf) +DEF_OP_V (roundf, 4, float, __builtin_roundf) +DEF_OP_V (roundf, 8, float, __builtin_roundf) +DEF_OP_V (roundf, 16, float, __builtin_roundf) +DEF_OP_V (roundf, 32, float, __builtin_roundf) +DEF_OP_V (roundf, 64, float, __builtin_roundf) +DEF_OP_V (roundf, 128, float, __builtin_roundf) +DEF_OP_V (roundf, 256, float, __builtin_roundf) +DEF_OP_V (roundf, 512, float, __builtin_roundf) +DEF_OP_V (roundf, 1024, float, __builtin_roundf) + +DEF_OP_V (round, 1, double, __builtin_round) +DEF_OP_V (round, 2, double, __builtin_round) +DEF_OP_V (round, 4, double, __builtin_round) +DEF_OP_V (round, 8, double, __builtin_round) +DEF_OP_V (round, 16, double, __builtin_round) +DEF_OP_V (round, 32, double, __builtin_round) +DEF_OP_V (round, 64, double, __builtin_round) +DEF_OP_V (round, 128, double, __builtin_round) +DEF_OP_V (round, 256, double, __builtin_round) +DEF_OP_V (round, 512, double, __builtin_round) + +/* { dg-final { scan-assembler-not {csrr} } } */ +/* { dg-final { scan-tree-dump-not "1,1" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "2,2" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "4,4" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "16,16" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "32,32" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "64,64" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "128,128" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "256,256" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "512,512" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "1024,1024" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "2048,2048" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "4096,4096" "optimized" } } */ +/* { dg-final { scan-assembler-times {vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t} 30 } } */ +/* { dg-final { scan-assembler-times {vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t} 30 } } */
Committed, thanks Juzhe.
Pan
From: juzhe.zhong@rivai.ai <juzhe.zhong@rivai.ai>
Sent: Tuesday, September 26, 2023 7:08 PM
To: Li, Pan2 <pan2.li@intel.com>; gcc-patches <gcc-patches@gcc.gnu.org>
Cc: Li, Pan2 <pan2.li@intel.com>; Wang, Yanzhang <yanzhang.wang@intel.com>; kito.cheng <kito.cheng@gmail.com>
Subject: Re: [PATCH v1] RISC-V: Support FP round auto-vectorization
LGTM
diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index 1d2fca60e98..798cf1272c5 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -2251,3 +2251,13 @@ (define_expand "rint<mode>2" DONE; } ) + +(define_expand "round<mode>2" + [(match_operand:V_VLSF 0 "register_operand") + (match_operand:V_VLSF 1 "register_operand")] + "TARGET_VECTOR && !flag_trapping_math && !flag_rounding_math" + { + riscv_vector::expand_vec_round (operands[0], operands[1], <MODE>mode, <VCONVERT>mode); + DONE; + } +) diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h index 629adeea94c..70ca244c591 100644 --- a/gcc/config/riscv/riscv-protos.h +++ b/gcc/config/riscv/riscv-protos.h @@ -256,6 +256,9 @@ enum insn_flags : unsigned int /* Means INSN has FRM operand and the value is FRM_RDN. */ FRM_RDN_P = 1 << 17, + + /* Means INSN has FRM operand and the value is FRM_RMM. */ + FRM_RMM_P = 1 << 18, }; enum insn_type : unsigned int @@ -299,6 +302,7 @@ enum insn_type : unsigned int UNARY_OP_TAMU_FRM_DYN = UNARY_OP_TAMU | FRM_DYN_P, UNARY_OP_TAMU_FRM_RUP = UNARY_OP_TAMU | FRM_RUP_P, UNARY_OP_TAMU_FRM_RDN = UNARY_OP_TAMU | FRM_RDN_P, + UNARY_OP_TAMU_FRM_RMM = UNARY_OP_TAMU | FRM_RMM_P, /* Binary operator. */ BINARY_OP = __NORMAL_OP | BINARY_OP_P, @@ -463,6 +467,7 @@ void expand_vec_ceil (rtx, rtx, machine_mode, machine_mode); void expand_vec_floor (rtx, rtx, machine_mode, machine_mode); void expand_vec_nearbyint (rtx, rtx, machine_mode, machine_mode); void expand_vec_rint (rtx, rtx, machine_mode, machine_mode); +void expand_vec_round (rtx, rtx, machine_mode, machine_mode); #endif bool sew64_scalar_helper (rtx *, rtx *, rtx, machine_mode, bool, void (*)(rtx *, rtx)); diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc index 445ed000f88..5f738634219 100644 --- a/gcc/config/riscv/riscv-v.cc +++ b/gcc/config/riscv/riscv-v.cc @@ -330,6 +330,8 @@ public: add_rounding_mode_operand (FRM_RUP); else if (m_insn_flags & FRM_RDN_P) add_rounding_mode_operand (FRM_RDN); + else if (m_insn_flags & FRM_RMM_P) + add_rounding_mode_operand (FRM_RMM); gcc_assert (insn_data[(int) icode].n_operands == m_opno); expand (icode, any_mem_p); @@ -3720,4 +3722,26 @@ expand_vec_rint (rtx op_0, rtx op_1, machine_mode vec_fp_mode, emit_vec_copysign (op_0, op_0, op_1, vec_fp_mode); } +void +expand_vec_round (rtx op_0, rtx op_1, machine_mode vec_fp_mode, + machine_mode vec_int_mode) +{ + /* Step-1: Get the abs float value for mask generation. */ + emit_vec_abs (op_0, op_1, vec_fp_mode); + + /* Step-2: Generate the mask on const fp. */ + rtx const_fp = get_fp_rounding_coefficient (GET_MODE_INNER (vec_fp_mode)); + rtx mask = emit_vec_float_cmp_mask (op_0, LT, const_fp, vec_fp_mode); + + /* Step-3: Convert to integer on mask, rounding to nearest (aka round). */ + rtx tmp = gen_reg_rtx (vec_int_mode); + emit_vec_cvt_x_f (tmp, op_1, mask, UNARY_OP_TAMU_FRM_RMM, vec_fp_mode); + + /* Step-4: Convert to floating-point on mask for the round result. */ + emit_vec_cvt_f_x (op_0, tmp, mask, UNARY_OP_TAMU_FRM_RMM, vec_fp_mode); + + /* Step-5: Retrieve the sign bit for -0.0. */ + emit_vec_copysign (op_0, op_0, op_1, vec_fp_mode); +} + } // namespace riscv_vector diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c new file mode 100644 index 00000000000..06de57bf7e2 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-0.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv_zvfh -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test__Float16___builtin_roundf16: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e16,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (_Float16, __builtin_roundf16) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c new file mode 100644 index 00000000000..ee51bcd820b --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-1.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_float___builtin_roundf: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e32,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (float, __builtin_roundf) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c new file mode 100644 index 00000000000..d78f0583e41 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-2.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_double___builtin_round: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e64,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_UNARY_CALL (double, __builtin_round) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c new file mode 100644 index 00000000000..98d14673e20 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-3.c @@ -0,0 +1,25 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math -fno-schedule-insns -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "test-math.h" + +/* +** test_float___builtin_roundf: +** frrm\s+[atx][0-9]+ +** ... +** fsrmi\s+4 +** ... +** vsetvli\s+[atx][0-9]+,\s*zero,\s*e32,\s*m1,\s*ta,\s*mu +** vfabs\.v\s+v[0-9]+,\s*v[0-9]+ +** vmflt\.vf\s+v0,\s*v[0-9]+,\s*[fa]+[0-9]+ +** vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t +** vfsgnj\.vv\s+v[0-9]+,v[0-9]+,v[0-9]+ +** ... +** vmerge\.vvm\s+v[0-9]+,\s*v[0-9]+,\s*v[0-9]+,\s*v0 +** ... +** fsrm\s+[atx][0-9]+ +** ... +*/ +TEST_COND_UNARY_CALL (float, __builtin_roundf) diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c new file mode 100644 index 00000000000..baaa614879a --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-1.c @@ -0,0 +1,39 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "-std=c99 -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math" } */ + +#include "test-math.h" + +#define ARRAY_SIZE 128 + +float in[ARRAY_SIZE]; +float out[ARRAY_SIZE]; +float ref[ARRAY_SIZE]; + +TEST_UNARY_CALL (float, __builtin_roundf) +TEST_ASSERT (float) + +TEST_INIT (float, 1.2, 1.0, 1) +TEST_INIT (float, -1.6, -2.0, 2) +TEST_INIT (float, 3.0, 3.0, 3) +TEST_INIT (float, 8388607.5, 8388608.0, 4) +TEST_INIT (float, 8388609.0, 8388609.0, 5) +TEST_INIT (float, 0.0, 0.0, 6) +TEST_INIT (float, -0.0, -0.0, 7) +TEST_INIT (float, -8388607.5, -8388608.0, 8) +TEST_INIT (float, -8388608.0, -8388608.0, 9) + +int +main () +{ + RUN_TEST (float, 1, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 2, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 3, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 4, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 5, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 6, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 7, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 8, __builtin_roundf, in, out, ref, ARRAY_SIZE); + RUN_TEST (float, 9, __builtin_roundf, in, out, ref, ARRAY_SIZE); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c new file mode 100644 index 00000000000..8ed82525814 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/unop/math-round-run-2.c @@ -0,0 +1,39 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "-std=c99 -O3 -ftree-vectorize -fno-vect-cost-model -ffast-math" } */ + +#include "test-math.h" + +#define ARRAY_SIZE 128 + +double in[ARRAY_SIZE]; +double out[ARRAY_SIZE]; +double ref[ARRAY_SIZE]; + +TEST_UNARY_CALL (double, __builtin_round) +TEST_ASSERT (double) + +TEST_INIT (double, 1.2, 1.0, 1) +TEST_INIT (double, -1.8, -2.0, 2) +TEST_INIT (double, 3.0, 3.0, 3) +TEST_INIT (double, 4503599627370495.5, 4503599627370496.0, 4) +TEST_INIT (double, 4503599627370497.0, 4503599627370497.0, 5) +TEST_INIT (double, 0.0, 0.0, 6) +TEST_INIT (double, -0.0, -0.0, 7) +TEST_INIT (double, -4503599627370495.5, -4503599627370496.0, 8) +TEST_INIT (double, -4503599627370496.0, -4503599627370496.0, 9) + +int +main () +{ + RUN_TEST (double, 1, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 2, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 3, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 4, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 5, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 6, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 7, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 8, __builtin_round, in, out, ref, ARRAY_SIZE); + RUN_TEST (double, 9, __builtin_round, in, out, ref, ARRAY_SIZE); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c new file mode 100644 index 00000000000..97fd6975969 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/vls/math-round-1.c @@ -0,0 +1,56 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv_zvfh_zvl4096b -mabi=lp64d -O3 --param=riscv-autovec-lmul=m8 -ffast-math -fdump-tree-optimized" } */ + +#include "def.h" + +DEF_OP_V (roundf16, 1, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 2, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 4, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 8, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 16, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 32, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 64, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 128, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 256, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 512, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 1024, _Float16, __builtin_roundf16) +DEF_OP_V (roundf16, 2048, _Float16, __builtin_roundf16) + +DEF_OP_V (roundf, 1, float, __builtin_roundf) +DEF_OP_V (roundf, 2, float, __builtin_roundf) +DEF_OP_V (roundf, 4, float, __builtin_roundf) +DEF_OP_V (roundf, 8, float, __builtin_roundf) +DEF_OP_V (roundf, 16, float, __builtin_roundf) +DEF_OP_V (roundf, 32, float, __builtin_roundf) +DEF_OP_V (roundf, 64, float, __builtin_roundf) +DEF_OP_V (roundf, 128, float, __builtin_roundf) +DEF_OP_V (roundf, 256, float, __builtin_roundf) +DEF_OP_V (roundf, 512, float, __builtin_roundf) +DEF_OP_V (roundf, 1024, float, __builtin_roundf) + +DEF_OP_V (round, 1, double, __builtin_round) +DEF_OP_V (round, 2, double, __builtin_round) +DEF_OP_V (round, 4, double, __builtin_round) +DEF_OP_V (round, 8, double, __builtin_round) +DEF_OP_V (round, 16, double, __builtin_round) +DEF_OP_V (round, 32, double, __builtin_round) +DEF_OP_V (round, 64, double, __builtin_round) +DEF_OP_V (round, 128, double, __builtin_round) +DEF_OP_V (round, 256, double, __builtin_round) +DEF_OP_V (round, 512, double, __builtin_round) + +/* { dg-final { scan-assembler-not {csrr} } } */ +/* { dg-final { scan-tree-dump-not "1,1" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "2,2" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "4,4" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "16,16" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "32,32" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "64,64" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "128,128" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "256,256" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "512,512" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "1024,1024" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "2048,2048" "optimized" } } */ +/* { dg-final { scan-tree-dump-not "4096,4096" "optimized" } } */ +/* { dg-final { scan-assembler-times {vfcvt\.x\.f\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t} 30 } } */ +/* { dg-final { scan-assembler-times {vfcvt\.f\.x\.v\s+v[0-9]+,\s*v[0-9]+,\s*v0\.t} 30 } } */