From patchwork Tue Nov 24 16:30:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nathan Sidwell X-Patchwork-Id: 1405659 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=acm.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=aVuNz4lQ; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CgTzv5HZxz9sRR for ; Wed, 25 Nov 2020 03:31:22 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DA4133948495; Tue, 24 Nov 2020 16:31:19 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) by sourceware.org (Postfix) with ESMTPS id 7403C3851C39 for ; Tue, 24 Nov 2020 16:30:57 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 7403C3851C39 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=acm.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=nathanmsidwell@gmail.com Received: by mail-qt1-x82c.google.com with SMTP id t5so16520133qtp.2 for ; Tue, 24 Nov 2020 08:30:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:to:from:subject:message-id:date:user-agent:mime-version :content-language; bh=UJVA2xbUnLzVJCQcJxMvYpAREUHhlMfW/ISr9XGH6mE=; b=aVuNz4lQjTVBIghjk3eQyEtPqVDkhut9tR0TGImmGT3ptfsiPalJhGwc4mX4yWzPle 4LVxI8dy6zjlr59PKrERdv2fRMKTOZZSH68OAzIb0SM4I5NuBIdbheU7OiTUoBmxkMfO 8+f8OBpKGZrt1jmdY48od2EHtKRkPbkuQZmKa/x09vzeCehahL5VitdDSLlynVi84u9L AXnoSRZcHPS0HLk51RKzfHCJeMqsca7lkzFfpUOwpbUEjzNdQV2z8eicnukN0d/vELaw seStD+FUwQjAv8H3lwNMEV3TiIjT70y5utIeZSPODo/o7FGwOMPUD6qpnncy47B00sgO 2iOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:to:from:subject:message-id:date :user-agent:mime-version:content-language; bh=UJVA2xbUnLzVJCQcJxMvYpAREUHhlMfW/ISr9XGH6mE=; b=en5Mvp1s3KY/GnwjC9yubanqkIDOOV1cIV1uPAljFk/Pen5tDWGeGMASvFLTHMBQLy KNoakU7UASHmnaQfjn/iBbYOjBNHU6H0UinP4JaLUq6pQNKsEBs26EOXjfw5cGCoARLZ xbMQZBaK2TBN2nwjocyU20uIgIyVu6rWxYsed3TYCZtfcxRwNQNXs7S7ssvRvw3CC8HC j6WsOW3qJq0fsvbKACf4MOSDyJqVAHoBSxQJOZZ1o2i7qmP03BCwMxlTAZlA80jFWSeV ozoPoijrSU5n0cKMKDbqRO6xxEVzXGU+kJCaQUyb4336HHzv7FmBqSo5xvWB780vs3+4 s0HQ== X-Gm-Message-State: AOAM533fgRlDDzKwu+UEjH8Gsp6OtqedNyniB1U05k5avOwGFQZlKCqV zDXu3F4XO6UQen3tFinTbtmOqYPkZtg= X-Google-Smtp-Source: ABdhPJxfgtrjtmvZpMqO7wKLiz24SuO4rMBxjvZ8ivWLqXtbEUYa+tGcTRIajSUXA1ItB61L+0k0JQ== X-Received: by 2002:aed:20c2:: with SMTP id 60mr5170007qtb.280.1606235456617; Tue, 24 Nov 2020 08:30:56 -0800 (PST) Received: from ?IPv6:2620:10d:c0a8:1102:4232:e32:2971:cf? ([2620:10d:c091:480::1:63c4]) by smtp.googlemail.com with ESMTPSA id h26sm13315837qkh.127.2020.11.24.08.30.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Nov 2020 08:30:54 -0800 (PST) To: GCC Patches From: Nathan Sidwell Subject: preprocessor: Add deferred macros Message-ID: <75547dcf-ca26-0a65-2501-116a6ba69641@acm.org> Date: Tue, 24 Nov 2020 11:30:53 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 Content-Language: en-US X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces@gcc.gnu.org Sender: "Gcc-patches" this completes the libcpp changes for C++ modules. It is slightly different to the patch I posted earlier, as I discovered I'd added a bitfield I didn't end up using. so that's now removed. Deferred macros are needed for C++ modules. Header units may export macro definitions and undefinitions. These are resolved lazily at the point of (potential) use. (The language specifies that, it's not just a useful optimization.) Thus, identifier nodes grow a 'deferred' field, which fortunately doesn't expand the structure on 64-bit systems as there was padding there. This is non-zero on NT_MACRO nodes, if the macro is deferred. When such an identifier is lexed, it is resolved via a callback that I added recently. That will either provide the macro definition, or discover it there was an overriding undef. Either way the identifier is no longer a deferred macro. Notice it is now possible for NT_MACRO nodes to have a NULL macro expansion. libcpp/ * include/cpplib.h (struct cpp_hashnode): Add deferred field. (cpp_set_deferred_macro): Define. (cpp_get_deferred_macro): Declare. (cpp_macro_definition): Reformat, add overload. (cpp_macro_definition_location): Deal with deferred macro. (cpp_alloc_token_string, cpp_compare_macro): Declare. * internal.h (_cpp_notify_macro_use): Return bool (_cpp_maybe_notify_macro_use): Likewise. * directives.c (do_undef): Check macro is not undef before warning. (do_ifdef, do_ifndef): Deal with deferred macro. * expr.c (parse_defined): Likewise. * lex.c (cpp_allocate_token_string): Break out of ... (create_literal): ... here. Call it. (cpp_maybe_module_directive): Deal with deferred macro. * macro.c (cpp_get_token_1): Deal with deferred macro. (warn_of_redefinition): Deal with deferred macro. (compare_macros): Rename to ... (cpp_compare_macro): ... here. Make extern. (cpp_get_deferred_macro): New. (_cpp_notify_macro_use): Deal with deferred macro, return bool indicating definedness. (cpp_macro_definition): Deal with deferred macro. pushing to trunk nathan diff --git i/libcpp/directives.c w/libcpp/directives.c index bffdc913adb..fa66b5c5f71 100644 --- i/libcpp/directives.c +++ w/libcpp/directives.c @@ -667,7 +667,8 @@ do_undef (cpp_reader *pfile) pfile->directive_line, 0, "undefining \"%s\"", NODE_NAME (node)); - if (CPP_OPTION (pfile, warn_unused_macros)) + if (node->value.macro + && CPP_OPTION (pfile, warn_unused_macros)) _cpp_warn_if_unused_macro (pfile, node, NULL); _cpp_free_definition (node); @@ -1981,8 +1982,10 @@ do_ifdef (cpp_reader *pfile) if (node) { skip = !_cpp_defined_macro_p (node); + if (!_cpp_maybe_notify_macro_use (pfile, node, pfile->directive_line)) + /* It wasn't a macro after all. */ + skip = true; _cpp_mark_macro_used (node); - _cpp_maybe_notify_macro_use (pfile, node, pfile->directive_line); if (pfile->cb.used) pfile->cb.used (pfile, pfile->directive_line, node); check_eol (pfile, false); @@ -2006,8 +2009,10 @@ do_ifndef (cpp_reader *pfile) if (node) { skip = _cpp_defined_macro_p (node); + if (!_cpp_maybe_notify_macro_use (pfile, node, pfile->directive_line)) + /* It wasn't a macro after all. */ + skip = false; _cpp_mark_macro_used (node); - _cpp_maybe_notify_macro_use (pfile, node, pfile->directive_line); if (pfile->cb.used) pfile->cb.used (pfile, pfile->directive_line, node); check_eol (pfile, false); diff --git i/libcpp/expr.c w/libcpp/expr.c index b98c0386eb5..2ba7726d61c 100644 --- i/libcpp/expr.c +++ w/libcpp/expr.c @@ -1068,6 +1068,7 @@ parse_defined (cpp_reader *pfile) } } + bool is_defined = false; if (node) { if ((pfile->context != initial_context @@ -1075,9 +1076,11 @@ parse_defined (cpp_reader *pfile) && CPP_OPTION (pfile, warn_expansion_to_defined)) cpp_pedwarning (pfile, CPP_W_EXPANSION_TO_DEFINED, "this use of \"defined\" may not be portable"); - + is_defined = _cpp_defined_macro_p (node); + if (!_cpp_maybe_notify_macro_use (pfile, node, token->src_loc)) + /* It wasn't a macro after all. */ + is_defined = false; _cpp_mark_macro_used (node); - _cpp_maybe_notify_macro_use (pfile, node, token->src_loc); /* A possible controlling macro of the form #if !defined (). _cpp_parse_expr checks there was no other junk on the line. */ @@ -1093,7 +1096,7 @@ parse_defined (cpp_reader *pfile) result.unsignedp = false; result.high = 0; result.overflow = false; - result.low = node && _cpp_defined_macro_p (node); + result.low = is_defined; return result; } diff --git i/libcpp/include/cpplib.h w/libcpp/include/cpplib.h index 91226cfc248..2becd2e8e54 100644 --- i/libcpp/include/cpplib.h +++ w/libcpp/include/cpplib.h @@ -901,7 +901,7 @@ enum cpp_builtin_type union GTY(()) _cpp_hashnode_value { /* Assert (maybe NULL) */ cpp_macro * GTY((tag ("NT_VOID"))) answers; - /* Macro (never NULL) */ + /* Macro (maybe NULL) */ cpp_macro * GTY((tag ("NT_USER_MACRO"))) macro; /* Code for a builtin macro. */ enum cpp_builtin_type GTY ((tag ("NT_BUILTIN_MACRO"))) builtin; @@ -919,7 +919,11 @@ struct GTY(()) cpp_hashnode { unsigned int flags : 9; /* CPP flags. */ ENUM_BITFIELD(node_type) type : 2; /* CPP node type. */ - /* 5 bits spare (plus another 32 on 64-bit hosts). */ + /* 5 bits spare. */ + + /* On a 64-bit system there would be 32-bits of padding to the value + field. So placing the deferred index here is not costly. */ + unsigned deferred; /* Deferred index, (unless zero). */ union _cpp_hashnode_value GTY ((desc ("%1.type"))) value; }; @@ -1061,6 +1065,18 @@ inline bool cpp_macro_p (const cpp_hashnode *node) { return node->type & NT_MACRO_MASK; } +inline cpp_macro *cpp_set_deferred_macro (cpp_hashnode *node, + cpp_macro *forced = NULL) +{ + cpp_macro *old = node->value.macro; + + node->value.macro = forced; + node->type = NT_USER_MACRO; + node->flags &= ~NODE_USED; + + return old; +} +cpp_macro *cpp_get_deferred_macro (cpp_reader *, cpp_hashnode *, location_t); /* Returns true if NODE is a function-like user macro. */ inline bool cpp_fun_like_macro_p (cpp_hashnode *node) @@ -1068,11 +1084,13 @@ inline bool cpp_fun_like_macro_p (cpp_hashnode *node) return cpp_user_macro_p (node) && node->value.macro->fun_like; } -extern const unsigned char *cpp_macro_definition (cpp_reader *, - cpp_hashnode *); +extern const unsigned char *cpp_macro_definition (cpp_reader *, cpp_hashnode *); +extern const unsigned char *cpp_macro_definition (cpp_reader *, cpp_hashnode *, + const cpp_macro *); inline location_t cpp_macro_definition_location (cpp_hashnode *node) { - return node->value.macro->line; + const cpp_macro *macro = node->value.macro; + return macro ? macro->line : 0; } /* Return an idempotent time stamp (possibly from SOURCE_DATE_EPOCH). */ enum class CPP_time_kind @@ -1266,6 +1284,8 @@ extern int cpp_ideq (const cpp_token *, const char *); extern void cpp_output_line (cpp_reader *, FILE *); extern unsigned char *cpp_output_line_to_string (cpp_reader *, const unsigned char *); +extern const unsigned char *cpp_alloc_token_string + (cpp_reader *, const unsigned char *, unsigned); extern void cpp_output_token (const cpp_token *, FILE *); extern const char *cpp_type2name (enum cpp_ttype, unsigned char flags); /* Returns the value of an escape sequence, truncated to the correct @@ -1321,6 +1341,8 @@ extern void cpp_scan_nooutput (cpp_reader *); extern int cpp_sys_macro_p (cpp_reader *); extern unsigned char *cpp_quote_string (unsigned char *, const unsigned char *, unsigned int); +extern bool cpp_compare_macros (const cpp_macro *macro1, + const cpp_macro *macro2); /* In files.c */ extern bool cpp_included (cpp_reader *, const char *); diff --git i/libcpp/internal.h w/libcpp/internal.h index 697fef053ba..45bbbddf268 100644 --- i/libcpp/internal.h +++ w/libcpp/internal.h @@ -662,13 +662,14 @@ inline bool _cpp_defined_macro_p (cpp_hashnode *node) } /* In macro.c */ -extern void _cpp_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, - location_t loc); -inline void _cpp_maybe_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, +extern bool _cpp_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, + location_t); +inline bool _cpp_maybe_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, location_t loc) { if (!(node->flags & NODE_USED)) - _cpp_notify_macro_use (pfile, node, loc); + return _cpp_notify_macro_use (pfile, node, loc); + return true; } extern cpp_macro *_cpp_new_macro (cpp_reader *, cpp_macro_kind, void *); extern void _cpp_free_definition (cpp_hashnode *); diff --git i/libcpp/lex.c w/libcpp/lex.c index 0f18daf6764..07d5a4ff466 100644 --- i/libcpp/lex.c +++ w/libcpp/lex.c @@ -1577,13 +1577,20 @@ static void create_literal (cpp_reader *pfile, cpp_token *token, const uchar *base, unsigned int len, enum cpp_ttype type) { - uchar *dest = _cpp_unaligned_alloc (pfile, len + 1); - - memcpy (dest, base, len); - dest[len] = '\0'; token->type = type; token->val.str.len = len; - token->val.str.text = dest; + token->val.str.text = cpp_alloc_token_string (pfile, base, len); +} + +const uchar * +cpp_alloc_token_string (cpp_reader *pfile, + const unsigned char *ptr, unsigned len) +{ + uchar *dest = _cpp_unaligned_alloc (pfile, len + 1); + + dest[len] = 0; + memcpy (dest, ptr, len); + return dest; } /* A pair of raw buffer pointers. The currently open one is [1], the @@ -2712,6 +2719,7 @@ cpp_maybe_module_directive (cpp_reader *pfile, cpp_token *result) /* Don't attempt to expand the token. */ tok->flags |= NO_EXPAND; if (_cpp_defined_macro_p (node) + && _cpp_maybe_notify_macro_use (pfile, node, tok->src_loc) && !cpp_fun_like_macro_p (node)) cpp_error_with_line (pfile, CPP_DL_ERROR, tok->src_loc, 0, "module control-line \"%s\" cannot be" diff --git i/libcpp/macro.c w/libcpp/macro.c index 35a5e708519..05755859cd6 100644 --- i/libcpp/macro.c +++ w/libcpp/macro.c @@ -268,6 +268,8 @@ class vaopt_state { /* Macro expansion. */ +static cpp_macro *get_deferred_or_lazy_macro (cpp_reader *, cpp_hashnode *, + location_t); static int enter_macro_context (cpp_reader *, cpp_hashnode *, const cpp_token *, location_t); static int builtin_macro (cpp_reader *, cpp_hashnode *, @@ -338,10 +340,6 @@ static cpp_macro *create_iso_definition (cpp_reader *); /* #define directive parsing and handling. */ static cpp_macro *lex_expansion_token (cpp_reader *, cpp_macro *); -static bool warn_of_redefinition (cpp_reader *, cpp_hashnode *, - const cpp_macro *); -static bool compare_macros (const cpp_macro *, const cpp_macro *); - static bool parse_params (cpp_reader *, unsigned *, bool *); static void check_trad_stringification (cpp_reader *, const cpp_macro *, const cpp_string *); @@ -353,8 +351,6 @@ static const cpp_token* cpp_get_token_1 (cpp_reader *, location_t *); static cpp_hashnode* macro_of_context (cpp_context *context); -static bool in_macro_expansion_p (cpp_reader *pfile); - /* Statistical counter tracking the number of macros that got expanded. */ unsigned num_expanded_macros_counter = 0; @@ -2878,6 +2874,12 @@ cpp_get_token_1 (cpp_reader *pfile, location_t *location) if (node->type == NT_VOID || (result->flags & NO_EXPAND)) break; + if (!(node->flags & NODE_USED) + && node->type == NT_USER_MACRO + && !node->value.macro + && !cpp_get_deferred_macro (pfile, node, result->src_loc)) + break; + if (!(node->flags & NODE_DISABLED)) { int ret = 0; @@ -3216,22 +3218,15 @@ warn_of_redefinition (cpp_reader *pfile, cpp_hashnode *node, if (node->flags & NODE_CONDITIONAL) return false; - cpp_macro *macro1 = node->value.macro; - if (macro1->lazy) - { - /* We don't want to mark MACRO as used, but do need to finalize - its laziness. */ - pfile->cb.user_lazy_macro (pfile, macro1, macro1->lazy - 1); - macro1->lazy = 0; - } - - return compare_macros (macro1, macro2); + if (cpp_macro *macro1 = get_deferred_or_lazy_macro (pfile, node, macro2->line)) + return cpp_compare_macros (macro1, macro2); + return false; } /* Return TRUE if MACRO1 and MACRO2 differ. */ -static bool -compare_macros (const cpp_macro *macro1, const cpp_macro *macro2) +bool +cpp_compare_macros (const cpp_macro *macro1, const cpp_macro *macro2) { /* Redefinition of a macro is allowed if and only if the old and new definitions are the same. (6.10.3 paragraph 2). */ @@ -3790,11 +3785,46 @@ cpp_define_lazily (cpp_reader *pfile, cpp_hashnode *node, unsigned num) macro->lazy = num + 1; } +/* NODE is a deferred macro, resolve it, returning the definition + (which may be NULL). */ +cpp_macro * +cpp_get_deferred_macro (cpp_reader *pfile, cpp_hashnode *node, + location_t loc) +{ + node->value.macro = pfile->cb.user_deferred_macro (pfile, loc, node); + + if (!node->value.macro) + node->type = NT_VOID; + + return node->value.macro; +} + +static cpp_macro * +get_deferred_or_lazy_macro (cpp_reader *pfile, cpp_hashnode *node, + location_t loc) +{ + cpp_macro *macro = node->value.macro; + if (!macro) + { + macro = cpp_get_deferred_macro (pfile, node, loc); + if (!macro) + return NULL; + } + + if (macro->lazy) + { + pfile->cb.user_lazy_macro (pfile, macro, macro->lazy - 1); + macro->lazy = 0; + } + + return macro; +} + /* Notify the use of NODE in a macro-aware context (i.e. expanding it, or testing its existance). Also applies any lazy definition. Return FALSE if the macro isn't really there. */ -extern void +extern bool _cpp_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, location_t loc) { @@ -3802,14 +3832,8 @@ _cpp_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, switch (node->type) { case NT_USER_MACRO: - { - cpp_macro *macro = node->value.macro; - if (macro->lazy) - { - pfile->cb.user_lazy_macro (pfile, macro, macro->lazy - 1); - macro->lazy = 0; - } - } + if (!get_deferred_or_lazy_macro (pfile, node, loc)) + return false; /* FALLTHROUGH. */ case NT_BUILTIN_MACRO: @@ -3825,6 +3849,8 @@ _cpp_notify_macro_use (cpp_reader *pfile, cpp_hashnode *node, default: abort (); } + + return true; } /* Warn if a token in STRING matches one of a function-like MACRO's @@ -3877,12 +3903,19 @@ check_trad_stringification (cpp_reader *pfile, const cpp_macro *macro, const unsigned char * cpp_macro_definition (cpp_reader *pfile, cpp_hashnode *node) { - unsigned int i, len; - unsigned char *buffer; - gcc_checking_assert (cpp_user_macro_p (node)); - const cpp_macro *macro = node->value.macro; + if (const cpp_macro *macro = get_deferred_or_lazy_macro (pfile, node, 0)) + return cpp_macro_definition (pfile, node, macro); + return NULL; +} + +const unsigned char * +cpp_macro_definition (cpp_reader *pfile, cpp_hashnode *node, + const cpp_macro *macro) +{ + unsigned int i, len; + unsigned char *buffer; /* Calculate length. */ len = NODE_LEN (node) * 10 + 2; /* ' ' and NUL. */