From patchwork Mon May 13 02:27:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: liuhongt X-Patchwork-Id: 1934484 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=WmUqZDq8; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Vd3Nv0yflz1ymw for ; Mon, 13 May 2024 12:30:05 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 338A6384AB57 for ; Mon, 13 May 2024 02:30:03 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by sourceware.org (Postfix) with ESMTPS id 8C3C83858CDB for ; Mon, 13 May 2024 02:29:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8C3C83858CDB Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8C3C83858CDB Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.198.163.8 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715567383; cv=none; b=Pk8F2q0ruIhDaJzsuv17mcB5g2kz9A7moYIZAPjvmKaqNoVKGi6HdFKowdxATz5rFSUuKz1LJCrjstxQ7ZQ9dMsVpsoMmp/WFFHOynyoVLJE540JVkDFy9uSYcOQ2iKrTPle6Ii2Ju/nHm1BfPZwlld/dMWBufU/97rN+zitaSc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715567383; c=relaxed/simple; bh=MHkz9unwWcAQPbek8ORCv5S5h1AYGXyxruFGB4Vl4mo=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=KVw3EF3RbSHv9jwMMs5PpY7s2UIA6VH5uULP/2EI2G8JrmnjkHmhw8tUwESjh6EUJZhBTKTuHjHo4axNJzqrxTzyAAeB9l6/NR6EwoYiGQxWQMYnGA5mvWddCunGUcZyGYnZrqe96pDEFizCZRgFckl3m+3kEeWPjwQIf2dEKbo= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715567381; x=1747103381; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=MHkz9unwWcAQPbek8ORCv5S5h1AYGXyxruFGB4Vl4mo=; b=WmUqZDq8QYRdKHcpaCxCElpQnFIM+bH/9pwBeTZsiGDCQaoDUtGDNdqk 6z4gecvlKdg+/9Tj1YB3iTFD7Xv875R0JmD4G5nD84+AFsYWgdsXSohgW hzyuvS2EMCm42mdoT8mVfa2QlFKupJ0q7XaYrJvrbP4g68gZHMHoKrEtY zHBiqaMRNlEIAHCJQCHtuiBO63pAl1NYSSyzrmDZoBKN6Pindg+tY+9tP sQUcmK4O2hxjD2xKEoLj7F22zu+38GVCh+Uj72eZqP6YoggfeauzZw46K JRLQaI/WU+AiJtVC7Qv6HlPVJsCy9xDe2OxHDeVX/gbKhu/gLonQ9ryOH w==; X-CSE-ConnectionGUID: AWt5AL++SGOjlgY6citn8g== X-CSE-MsgGUID: 5qdKhH0dSq6xwoWPmFFKLw== X-IronPort-AV: E=McAfee;i="6600,9927,11071"; a="28999473" X-IronPort-AV: E=Sophos;i="6.08,157,1712646000"; d="scan'208";a="28999473" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 May 2024 19:29:40 -0700 X-CSE-ConnectionGUID: 6TPC3ZgaS/aOUdZy0edLXQ== X-CSE-MsgGUID: kCteQkKuQeO2IrREedq0dw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,157,1712646000"; d="scan'208";a="30756626" Received: from shvmail03.sh.intel.com ([10.239.245.20]) by orviesa007.jf.intel.com with ESMTP; 12 May 2024 19:29:39 -0700 Received: from shliclel4217.sh.intel.com (shliclel4217.sh.intel.com [10.239.240.127]) by shvmail03.sh.intel.com (Postfix) with ESMTP id 0EB3C10081C6; Mon, 13 May 2024 10:29:38 +0800 (CST) From: liuhongt To: gcc-patches@gcc.gnu.org Cc: richard.guenther@gmail.com Subject: [PATCH] Don't reduce estimated unrolled size for innermost loop. Date: Mon, 13 May 2024 10:27:37 +0800 Message-Id: <20240513022737.3105192-1-hongtao.liu@intel.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org As testcase in the PR, O3 cunrolli may prevent vectorization for the innermost loop and increase register pressure. The patch removes the 1/3 reduction of unr_insn for innermost loop for UL_ALL. ul != UR_ALL is needed since some small loop complete unrolling at O2 relies the reduction. Bootstrapped and regtested on x86_64-pc-linux-gnu{-m32,}. No big impact for SPEC2017. Ok for trunk? gcc/ChangeLog: PR tree-optimization/112325 * tree-ssa-loop-ivcanon.cc (estimated_unrolled_size): Add 2 new parameters: loop and ul, and remove unr_insns reduction for innermost loop. (try_unroll_loop_completely): Pass loop and ul to estimated_unrolled_size. gcc/testsuite/ChangeLog: * gcc.dg/tree-ssa/pr112325.c: New test. * gcc.dg/vect/pr69783.c: Add extra option --param max-completely-peeled-insns=300. --- gcc/testsuite/gcc.dg/tree-ssa/pr112325.c | 57 ++++++++++++++++++++++++ gcc/testsuite/gcc.dg/vect/pr69783.c | 2 +- gcc/tree-ssa-loop-ivcanon.cc | 16 +++++-- 3 files changed, 71 insertions(+), 4 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/pr112325.c diff --git a/gcc/testsuite/gcc.dg/tree-ssa/pr112325.c b/gcc/testsuite/gcc.dg/tree-ssa/pr112325.c new file mode 100644 index 00000000000..14208b3e7f8 --- /dev/null +++ b/gcc/testsuite/gcc.dg/tree-ssa/pr112325.c @@ -0,0 +1,57 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-cunrolli-details" } */ + +typedef unsigned short ggml_fp16_t; +static float table_f32_f16[1 << 16]; + +inline static float ggml_lookup_fp16_to_fp32(ggml_fp16_t f) { + unsigned short s; + __builtin_memcpy(&s, &f, sizeof(unsigned short)); + return table_f32_f16[s]; +} + +typedef struct { + ggml_fp16_t d; + ggml_fp16_t m; + unsigned char qh[4]; + unsigned char qs[32 / 2]; +} block_q5_1; + +typedef struct { + float d; + float s; + char qs[32]; +} block_q8_1; + +void ggml_vec_dot_q5_1_q8_1(const int n, float * restrict s, const void * restrict vx, const void * restrict vy) { + const int qk = 32; + const int nb = n / qk; + + const block_q5_1 * restrict x = vx; + const block_q8_1 * restrict y = vy; + + float sumf = 0.0; + + for (int i = 0; i < nb; i++) { + unsigned qh; + __builtin_memcpy(&qh, x[i].qh, sizeof(qh)); + + int sumi = 0; + + for (int j = 0; j < qk/2; ++j) { + const unsigned char xh_0 = ((qh >> (j + 0)) << 4) & 0x10; + const unsigned char xh_1 = ((qh >> (j + 12)) ) & 0x10; + + const int x0 = (x[i].qs[j] & 0xF) | xh_0; + const int x1 = (x[i].qs[j] >> 4) | xh_1; + + sumi += (x0 * y[i].qs[j]) + (x1 * y[i].qs[j + qk/2]); + } + + sumf += (ggml_lookup_fp16_to_fp32(x[i].d)*y[i].d)*sumi + ggml_lookup_fp16_to_fp32(x[i].m)*y[i].s; + } + + *s = sumf; +} + +/* { dg-final { scan-tree-dump {(?n)Not unrolling loop [1-9] \(--param max-completely-peel-times limit reached} "cunrolli"} } */ diff --git a/gcc/testsuite/gcc.dg/vect/pr69783.c b/gcc/testsuite/gcc.dg/vect/pr69783.c index 5df95d0ce4e..a1f75514d72 100644 --- a/gcc/testsuite/gcc.dg/vect/pr69783.c +++ b/gcc/testsuite/gcc.dg/vect/pr69783.c @@ -1,6 +1,6 @@ /* { dg-do compile } */ /* { dg-require-effective-target vect_float } */ -/* { dg-additional-options "-Ofast -funroll-loops" } */ +/* { dg-additional-options "-Ofast -funroll-loops --param max-completely-peeled-insns=300" } */ #define NXX 516 #define NYY 516 diff --git a/gcc/tree-ssa-loop-ivcanon.cc b/gcc/tree-ssa-loop-ivcanon.cc index bf017137260..5e0eca647a1 100644 --- a/gcc/tree-ssa-loop-ivcanon.cc +++ b/gcc/tree-ssa-loop-ivcanon.cc @@ -444,7 +444,9 @@ tree_estimate_loop_size (class loop *loop, edge exit, edge edge_to_cancel, static unsigned HOST_WIDE_INT estimated_unrolled_size (struct loop_size *size, - unsigned HOST_WIDE_INT nunroll) + unsigned HOST_WIDE_INT nunroll, + enum unroll_level ul, + class loop* loop) { HOST_WIDE_INT unr_insns = ((nunroll) * (HOST_WIDE_INT) (size->overall @@ -453,7 +455,15 @@ estimated_unrolled_size (struct loop_size *size, unr_insns = 0; unr_insns += size->last_iteration - size->last_iteration_eliminated_by_peeling; - unr_insns = unr_insns * 2 / 3; + /* For innermost loop, loop body is not likely to be simplied as much as 1/3. + and may increase a lot of register pressure. + UL != UL_ALL is need to unroll small loop at O2. */ + class loop *loop_father = loop_outer (loop); + if (loop->inner || !loop_father + || loop_father->latch == EXIT_BLOCK_PTR_FOR_FN (cfun) + || ul != UL_ALL) + unr_insns = unr_insns * 2 / 3; + if (unr_insns <= 0) unr_insns = 1; @@ -837,7 +847,7 @@ try_unroll_loop_completely (class loop *loop, unsigned HOST_WIDE_INT ninsns = size.overall; unsigned HOST_WIDE_INT unr_insns - = estimated_unrolled_size (&size, n_unroll); + = estimated_unrolled_size (&size, n_unroll, ul, loop); if (dump_file && (dump_flags & TDF_DETAILS)) { fprintf (dump_file, " Loop size: %d\n", (int) ninsns);