From patchwork Tue Jul 20 20:55:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 1507760 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=Yeai4Lb4; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GTrk70jTvz9sSs for ; Wed, 21 Jul 2021 07:01:50 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 6B0DE398AC35 for ; Tue, 20 Jul 2021 21:01:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6B0DE398AC35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1626814908; bh=aGNkAxT9f3JAAfmzLPISibbFnnoR23dDE4N/yQe0qtM=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Yeai4Lb4kfYkFHA4oHmdwtJ9bXkQ3Tg3LW8n3uDjgL0kvh35GVxIEwPJ/l85G+oev IPE44VqMbluPEDj7qTt3AgPzaLtqyJa9OBy654bpgJ95EgYLbwlvkSD+USqvsToUf0 IG9tVZVV4u8uiyiUEvwvR353FnhEVSFLIqI0Lc1o= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by sourceware.org (Postfix) with ESMTPS id E4A093848416 for ; Tue, 20 Jul 2021 20:55:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org E4A093848416 Received: by mail-pj1-x102b.google.com with SMTP id my10so421955pjb.1 for ; Tue, 20 Jul 2021 13:55:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aGNkAxT9f3JAAfmzLPISibbFnnoR23dDE4N/yQe0qtM=; b=bWtfiN0XHbAnasL5PrUsmmqKCiN0LXGsgwwnf6boLpPnyB8oUhC4BZmF5fIbGus9Z5 UfZBLDpAuciSZf9JylFhhC1nuyC4P4sgrtPScrS9srvGwMUNQT4prfBR6NTWCOuVxBPW j8nDTDBAvfQCpwa4/xTDhxKP1fp5TLXFiXORD7jVRSHAnRhvHEoYQ7Ij7v+uxpv2c0JL ly0XedgrdhNOeEe6g3FRd2SiyWZ+RiMahjI4gxPPOqfBgjh+4FzsEWWy3LhbydnLt3Q+ GFu+GiInthv6uX/ZEzfFO7Gl9saaUwg/lbWzGPWV5q9Z8/zht6RxhGOpk0hw67orNvk+ LDOg== X-Gm-Message-State: AOAM533+sScm4u3wW8A9MBMjYn+RE0/vkBiuHLm3OHLB9GMNSMhC6Z8n x/vLf8DBsN9X9tizTAZJaZexOYQrpmw= X-Google-Smtp-Source: ABdhPJwX/6DxpJSiXjDKhhGQwK1/slhwFF0hNJ0HU6KaQ7ohRD+DwWN6HnaQDWFkbg5ST7eI4Iuiog== X-Received: by 2002:a17:90a:dc0a:: with SMTP id i10mr31857615pjv.136.1626814519863; Tue, 20 Jul 2021 13:55:19 -0700 (PDT) Received: from gnu-cfl-2.localdomain ([172.58.35.147]) by smtp.gmail.com with ESMTPSA id p34sm759674pfh.172.2021.07.20.13.55.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Jul 2021 13:55:19 -0700 (PDT) Received: from gnu-tgl-2.localdomain (gnu-tgl-2 [192.168.1.34]) by gnu-cfl-2.localdomain (Postfix) with ESMTPS id AD7DAC0484; Tue, 20 Jul 2021 13:55:16 -0700 (PDT) Received: from gnu-tgl-2.lan (localhost [IPv6:::1]) by gnu-tgl-2.localdomain (Postfix) with ESMTP id 7AF5B3004C6; Tue, 20 Jul 2021 13:55:06 -0700 (PDT) To: gcc-patches@gcc.gnu.org Subject: [PATCH 1/4] libsanitizer: Merge with upstream Date: Tue, 20 Jul 2021 13:55:03 -0700 Message-Id: <20210720205506.296273-2-hjl.tools@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210720205506.296273-1-hjl.tools@gmail.com> References: <20210720205506.296273-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3033.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Gcc-patches" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Jakub Jelinek Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" Merged revision: 7704fedfff6ef5676adb6415f3be0ac927d1a746 --- libsanitizer/MERGE | 2 +- libsanitizer/asan/Makefile.am | 1 - libsanitizer/asan/Makefile.in | 8 +- libsanitizer/asan/asan_allocator.cpp | 10 +- libsanitizer/asan/asan_errors.cpp | 1 - libsanitizer/asan/asan_fake_stack.cpp | 50 +++- libsanitizer/asan/asan_flags.cpp | 4 - libsanitizer/asan/asan_flags.inc | 3 +- libsanitizer/asan/asan_globals.cpp | 19 ++ libsanitizer/asan/asan_interceptors.cpp | 28 +- libsanitizer/asan/asan_interceptors.h | 11 +- .../asan/asan_interceptors_memintrinsics.cpp | 6 +- libsanitizer/asan/asan_interface.inc | 11 + libsanitizer/asan/asan_internal.h | 15 +- libsanitizer/asan/asan_malloc_linux.cpp | 36 +-- libsanitizer/asan/asan_malloc_local.h | 52 ---- libsanitizer/asan/asan_mapping.h | 31 +- libsanitizer/asan/asan_mapping_myriad.h | 85 ------ libsanitizer/asan/asan_new_delete.cpp | 20 +- libsanitizer/asan/asan_poisoning.cpp | 19 +- libsanitizer/asan/asan_poisoning.h | 3 - libsanitizer/asan/asan_rtems.cpp | 266 ----------------- libsanitizer/asan/asan_rtl.cpp | 47 ++- libsanitizer/asan/asan_shadow_setup.cpp | 11 +- libsanitizer/asan/asan_stack.cpp | 3 +- libsanitizer/asan/asan_thread.cpp | 45 +-- libsanitizer/asan/asan_thread.h | 17 +- libsanitizer/hwasan/Makefile.am | 2 + libsanitizer/hwasan/Makefile.in | 9 +- libsanitizer/hwasan/hwasan.cpp | 77 ++++- libsanitizer/hwasan/hwasan.h | 42 ++- .../hwasan/hwasan_allocation_functions.cpp | 172 +++++++++++ libsanitizer/hwasan/hwasan_allocator.cpp | 35 ++- libsanitizer/hwasan/hwasan_allocator.h | 7 +- libsanitizer/hwasan/hwasan_dynamic_shadow.cpp | 4 +- libsanitizer/hwasan/hwasan_fuchsia.cpp | 192 ++++++++++++ libsanitizer/hwasan/hwasan_interceptors.cpp | 182 +----------- libsanitizer/hwasan/hwasan_linux.cpp | 166 ++++------- libsanitizer/hwasan/hwasan_mapping.h | 17 +- libsanitizer/hwasan/hwasan_new_delete.cpp | 39 ++- libsanitizer/hwasan/hwasan_poisoning.cpp | 24 -- libsanitizer/hwasan/hwasan_report.cpp | 206 ++++++++----- libsanitizer/hwasan/hwasan_thread.cpp | 21 +- libsanitizer/hwasan/hwasan_thread.h | 11 +- libsanitizer/hwasan/hwasan_thread_list.cpp | 2 +- libsanitizer/hwasan/hwasan_thread_list.h | 8 +- .../include/sanitizer/dfsan_interface.h | 95 +++--- libsanitizer/interception/interception.h | 33 +- libsanitizer/lsan/lsan.cpp | 14 +- libsanitizer/lsan/lsan_common.h | 4 +- libsanitizer/lsan/lsan_thread.cpp | 7 +- libsanitizer/sanitizer_common/Makefile.am | 2 +- libsanitizer/sanitizer_common/Makefile.in | 19 +- .../sanitizer_common/sanitizer_addrhashmap.h | 106 +++---- .../sanitizer_common/sanitizer_allocator.cpp | 38 +-- .../sanitizer_allocator_combined.h | 4 +- .../sanitizer_allocator_local_cache.h | 19 +- .../sanitizer_allocator_primary32.h | 4 +- .../sanitizer_allocator_primary64.h | 170 +++++------ .../sanitizer_allocator_secondary.h | 8 +- .../sanitizer_common/sanitizer_common.cpp | 15 +- .../sanitizer_common/sanitizer_common.h | 23 +- .../sanitizer_common_interceptors.inc | 84 ++++-- .../sanitizer_common_libcdep.cpp | 4 +- .../sanitizer_common_nolibc.cpp | 1 - .../sanitizer_deadlock_detector1.cpp | 2 +- .../sanitizer_deadlock_detector2.cpp | 32 +- .../sanitizer_common/sanitizer_errno.h | 3 +- .../sanitizer_common/sanitizer_fuchsia.cpp | 33 +- .../sanitizer_common/sanitizer_fuchsia.h | 2 + .../sanitizer_common/sanitizer_libc.h | 3 +- .../sanitizer_common/sanitizer_libignore.cpp | 25 ++ .../sanitizer_common/sanitizer_libignore.h | 35 +-- .../sanitizer_common/sanitizer_linux.cpp | 46 ++- .../sanitizer_linux_libcdep.cpp | 41 ++- .../sanitizer_common/sanitizer_mac.cpp | 25 +- libsanitizer/sanitizer_common/sanitizer_mac.h | 20 -- .../sanitizer_common/sanitizer_mutex.cpp | 39 +++ .../sanitizer_common/sanitizer_mutex.h | 272 ++++++++++++++--- .../sanitizer_common/sanitizer_netbsd.cpp | 11 +- .../sanitizer_common/sanitizer_platform.h | 25 +- .../sanitizer_platform_interceptors.h | 12 +- .../sanitizer_platform_limits_linux.cpp | 7 +- .../sanitizer_platform_limits_posix.h | 12 +- .../sanitizer_posix_libcdep.cpp | 15 +- .../sanitizer_common/sanitizer_printf.cpp | 27 +- .../sanitizer_common/sanitizer_quarantine.h | 3 +- .../sanitizer_common/sanitizer_rtems.cpp | 281 ------------------ .../sanitizer_common/sanitizer_rtems.h | 20 -- .../sanitizer_common/sanitizer_solaris.cpp | 18 +- .../sanitizer_common/sanitizer_stacktrace.cpp | 17 +- .../sanitizer_common/sanitizer_stacktrace.h | 36 ++- .../sanitizer_stacktrace_libcdep.cpp | 185 ++++++++---- .../sanitizer_symbolizer_markup.cpp | 15 +- .../sanitizer_symbolizer_report.cpp | 22 +- .../sanitizer_symbolizer_rtems.h | 40 --- .../sanitizer_thread_registry.cpp | 29 +- .../sanitizer_thread_registry.h | 18 +- .../sanitizer_thread_safety.h | 42 +++ .../sanitizer_common/sanitizer_win.cpp | 26 +- libsanitizer/tsan/Makefile.am | 3 +- libsanitizer/tsan/Makefile.in | 8 +- libsanitizer/tsan/tsan_clock.cpp | 29 +- libsanitizer/tsan/tsan_defs.h | 16 - libsanitizer/tsan/tsan_interceptors.h | 16 +- libsanitizer/tsan/tsan_interceptors_mac.cpp | 9 +- libsanitizer/tsan/tsan_interceptors_posix.cpp | 41 ++- libsanitizer/tsan/tsan_interface.h | 3 +- libsanitizer/tsan/tsan_interface_ann.cpp | 6 +- libsanitizer/tsan/tsan_interface_atomic.cpp | 54 ++-- libsanitizer/tsan/tsan_mman.cpp | 5 +- libsanitizer/tsan/tsan_mutex.cpp | 11 +- libsanitizer/tsan/tsan_mutex.h | 5 +- libsanitizer/tsan/tsan_platform.h | 54 ++++ libsanitizer/tsan/tsan_platform_linux.cpp | 6 + libsanitizer/tsan/tsan_platform_posix.cpp | 20 +- libsanitizer/tsan/tsan_rtl.cpp | 51 +--- libsanitizer/tsan/tsan_rtl.h | 21 -- libsanitizer/tsan/tsan_rtl_mutex.cpp | 17 -- libsanitizer/tsan/tsan_rtl_ppc64.S | 1 - libsanitizer/tsan/tsan_rtl_s390x.S | 47 +++ libsanitizer/tsan/tsan_rtl_thread.cpp | 12 - libsanitizer/tsan/tsan_stack_trace.cpp | 8 +- libsanitizer/tsan/tsan_stat.cpp | 186 ------------ libsanitizer/tsan/tsan_stat.h | 191 ------------ libsanitizer/tsan/tsan_sync.cpp | 5 +- libsanitizer/tsan/tsan_trace.h | 4 +- .../tsan/tsan_update_shadow_word_inl.h | 12 +- libsanitizer/ubsan/ubsan_diag_standalone.cpp | 8 +- libsanitizer/ubsan/ubsan_flags.cpp | 1 - libsanitizer/ubsan/ubsan_handlers.cpp | 15 - libsanitizer/ubsan/ubsan_handlers.h | 8 - libsanitizer/ubsan/ubsan_platform.h | 10 +- 133 files changed, 2284 insertions(+), 2708 deletions(-) delete mode 100644 libsanitizer/asan/asan_malloc_local.h delete mode 100644 libsanitizer/asan/asan_mapping_myriad.h delete mode 100644 libsanitizer/asan/asan_rtems.cpp create mode 100644 libsanitizer/hwasan/hwasan_allocation_functions.cpp create mode 100644 libsanitizer/hwasan/hwasan_fuchsia.cpp create mode 100644 libsanitizer/sanitizer_common/sanitizer_mutex.cpp delete mode 100644 libsanitizer/sanitizer_common/sanitizer_rtems.cpp delete mode 100644 libsanitizer/sanitizer_common/sanitizer_rtems.h delete mode 100644 libsanitizer/sanitizer_common/sanitizer_symbolizer_rtems.h create mode 100644 libsanitizer/sanitizer_common/sanitizer_thread_safety.h create mode 100644 libsanitizer/tsan/tsan_rtl_s390x.S delete mode 100644 libsanitizer/tsan/tsan_stat.cpp delete mode 100644 libsanitizer/tsan/tsan_stat.h diff --git a/libsanitizer/MERGE b/libsanitizer/MERGE index c4731d0866c..81d00f27de4 100644 --- a/libsanitizer/MERGE +++ b/libsanitizer/MERGE @@ -1,4 +1,4 @@ -f58e0513dd95944b81ce7a6e7b49ba656de7d75f +7704fedfff6ef5676adb6415f3be0ac927d1a746 The first line of this file holds the git revision number of the last merge done from the master library sources. diff --git a/libsanitizer/asan/Makefile.am b/libsanitizer/asan/Makefile.am index 2e6385509b4..74658ca7b9c 100644 --- a/libsanitizer/asan/Makefile.am +++ b/libsanitizer/asan/Makefile.am @@ -38,7 +38,6 @@ asan_files = \ asan_posix.cpp \ asan_premap_shadow.cpp \ asan_report.cpp \ - asan_rtems.cpp \ asan_rtl.cpp \ asan_shadow_setup.cpp \ asan_stack.cpp \ diff --git a/libsanitizer/asan/Makefile.in b/libsanitizer/asan/Makefile.in index 25c7fd7b759..53efe526f9c 100644 --- a/libsanitizer/asan/Makefile.in +++ b/libsanitizer/asan/Makefile.in @@ -156,9 +156,9 @@ am__objects_1 = asan_activation.lo asan_allocator.lo asan_debugging.lo \ asan_interceptors_memintrinsics.lo asan_linux.lo asan_mac.lo \ asan_malloc_linux.lo asan_malloc_mac.lo asan_malloc_win.lo \ asan_memory_profile.lo asan_new_delete.lo asan_poisoning.lo \ - asan_posix.lo asan_premap_shadow.lo asan_report.lo \ - asan_rtems.lo asan_rtl.lo asan_shadow_setup.lo asan_stack.lo \ - asan_stats.lo asan_suppressions.lo asan_thread.lo asan_win.lo \ + asan_posix.lo asan_premap_shadow.lo asan_report.lo asan_rtl.lo \ + asan_shadow_setup.lo asan_stack.lo asan_stats.lo \ + asan_suppressions.lo asan_thread.lo asan_win.lo \ asan_win_dll_thunk.lo asan_win_dynamic_runtime_thunk.lo \ asan_interceptors_vfork.lo am_libasan_la_OBJECTS = $(am__objects_1) @@ -446,7 +446,6 @@ asan_files = \ asan_posix.cpp \ asan_premap_shadow.cpp \ asan_report.cpp \ - asan_rtems.cpp \ asan_rtl.cpp \ asan_shadow_setup.cpp \ asan_stack.cpp \ @@ -604,7 +603,6 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_posix.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_premap_shadow.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_report.Plo@am__quote@ -@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_rtems.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_rtl.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_shadow_setup.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/asan_stack.Plo@am__quote@ diff --git a/libsanitizer/asan/asan_allocator.cpp b/libsanitizer/asan/asan_allocator.cpp index 7c8bb504332..414fba3b427 100644 --- a/libsanitizer/asan/asan_allocator.cpp +++ b/libsanitizer/asan/asan_allocator.cpp @@ -852,12 +852,12 @@ struct Allocator { quarantine.PrintStats(); } - void ForceLock() { + void ForceLock() ACQUIRE(fallback_mutex) { allocator.ForceLock(); fallback_mutex.Lock(); } - void ForceUnlock() { + void ForceUnlock() RELEASE(fallback_mutex) { fallback_mutex.Unlock(); allocator.ForceUnlock(); } @@ -1081,11 +1081,9 @@ uptr asan_mz_size(const void *ptr) { return instance.AllocationSize(reinterpret_cast(ptr)); } -void asan_mz_force_lock() { - instance.ForceLock(); -} +void asan_mz_force_lock() NO_THREAD_SAFETY_ANALYSIS { instance.ForceLock(); } -void asan_mz_force_unlock() { +void asan_mz_force_unlock() NO_THREAD_SAFETY_ANALYSIS { instance.ForceUnlock(); } diff --git a/libsanitizer/asan/asan_errors.cpp b/libsanitizer/asan/asan_errors.cpp index e68e6971f96..45166c06487 100644 --- a/libsanitizer/asan/asan_errors.cpp +++ b/libsanitizer/asan/asan_errors.cpp @@ -533,7 +533,6 @@ static void PrintLegend(InternalScopedString *str) { PrintShadowByte(str, " ASan internal: ", kAsanInternalHeapMagic); PrintShadowByte(str, " Left alloca redzone: ", kAsanAllocaLeftMagic); PrintShadowByte(str, " Right alloca redzone: ", kAsanAllocaRightMagic); - PrintShadowByte(str, " Shadow gap: ", kAsanShadowGap); } static void PrintShadowBytes(InternalScopedString *str, const char *before, diff --git a/libsanitizer/asan/asan_fake_stack.cpp b/libsanitizer/asan/asan_fake_stack.cpp index 1f873fec7d7..bf5c342ee59 100644 --- a/libsanitizer/asan/asan_fake_stack.cpp +++ b/libsanitizer/asan/asan_fake_stack.cpp @@ -187,7 +187,7 @@ void SetTLSFakeStack(FakeStack *fs) { } static FakeStack *GetFakeStack() { AsanThread *t = GetCurrentThread(); if (!t) return nullptr; - return t->fake_stack(); + return t->get_or_create_fake_stack(); } static FakeStack *GetFakeStackFast() { @@ -198,7 +198,13 @@ static FakeStack *GetFakeStackFast() { return GetFakeStack(); } -ALWAYS_INLINE uptr OnMalloc(uptr class_id, uptr size) { +static FakeStack *GetFakeStackFastAlways() { + if (FakeStack *fs = GetTLSFakeStack()) + return fs; + return GetFakeStack(); +} + +static ALWAYS_INLINE uptr OnMalloc(uptr class_id, uptr size) { FakeStack *fs = GetFakeStackFast(); if (!fs) return 0; uptr local_stack; @@ -210,7 +216,21 @@ ALWAYS_INLINE uptr OnMalloc(uptr class_id, uptr size) { return ptr; } -ALWAYS_INLINE void OnFree(uptr ptr, uptr class_id, uptr size) { +static ALWAYS_INLINE uptr OnMallocAlways(uptr class_id, uptr size) { + FakeStack *fs = GetFakeStackFastAlways(); + if (!fs) + return 0; + uptr local_stack; + uptr real_stack = reinterpret_cast(&local_stack); + FakeFrame *ff = fs->Allocate(fs->stack_size_log(), class_id, real_stack); + if (!ff) + return 0; // Out of fake stack. + uptr ptr = reinterpret_cast(ff); + SetShadow(ptr, size, class_id, 0); + return ptr; +} + +static ALWAYS_INLINE void OnFree(uptr ptr, uptr class_id, uptr size) { FakeStack::Deallocate(ptr, class_id); SetShadow(ptr, size, class_id, kMagic8); } @@ -219,14 +239,18 @@ ALWAYS_INLINE void OnFree(uptr ptr, uptr class_id, uptr size) { // ---------------------- Interface ---------------- {{{1 using namespace __asan; -#define DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(class_id) \ - extern "C" SANITIZER_INTERFACE_ATTRIBUTE uptr \ - __asan_stack_malloc_##class_id(uptr size) { \ - return OnMalloc(class_id, size); \ - } \ - extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __asan_stack_free_##class_id( \ - uptr ptr, uptr size) { \ - OnFree(ptr, class_id, size); \ +#define DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(class_id) \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE uptr \ + __asan_stack_malloc_##class_id(uptr size) { \ + return OnMalloc(class_id, size); \ + } \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE uptr \ + __asan_stack_malloc_always_##class_id(uptr size) { \ + return OnMallocAlways(class_id, size); \ + } \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE void __asan_stack_free_##class_id( \ + uptr ptr, uptr size) { \ + OnFree(ptr, class_id, size); \ } DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(0) @@ -240,7 +264,11 @@ DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(7) DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(8) DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(9) DEFINE_STACK_MALLOC_FREE_WITH_CLASS_ID(10) + extern "C" { +// TODO: remove this method and fix tests that use it by setting +// -asan-use-after-return=never, after modal UAR flag lands +// (https://github.com/google/sanitizers/issues/1394) SANITIZER_INTERFACE_ATTRIBUTE void *__asan_get_current_fake_stack() { return GetFakeStackFast(); } diff --git a/libsanitizer/asan/asan_flags.cpp b/libsanitizer/asan/asan_flags.cpp index cb6a89fe32c..c64e4647028 100644 --- a/libsanitizer/asan/asan_flags.cpp +++ b/libsanitizer/asan/asan_flags.cpp @@ -155,10 +155,6 @@ void InitializeFlags() { CHECK_LE(f->max_redzone, 2048); CHECK(IsPowerOfTwo(f->redzone)); CHECK(IsPowerOfTwo(f->max_redzone)); - if (SANITIZER_RTEMS) { - CHECK(!f->unmap_shadow_on_exit); - CHECK(!f->protect_shadow_gap); - } // quarantine_size is deprecated but we still honor it. // quarantine_size can not be used together with quarantine_size_mb. diff --git a/libsanitizer/asan/asan_flags.inc b/libsanitizer/asan/asan_flags.inc index 43c70dbca56..514b225c407 100644 --- a/libsanitizer/asan/asan_flags.inc +++ b/libsanitizer/asan/asan_flags.inc @@ -87,8 +87,7 @@ ASAN_FLAG(bool, check_malloc_usable_size, true, "295.*.") ASAN_FLAG(bool, unmap_shadow_on_exit, false, "If set, explicitly unmaps the (huge) shadow at exit.") -ASAN_FLAG(bool, protect_shadow_gap, !SANITIZER_RTEMS, - "If set, mprotect the shadow gap") +ASAN_FLAG(bool, protect_shadow_gap, true, "If set, mprotect the shadow gap") ASAN_FLAG(bool, print_stats, false, "Print various statistics after printing an error message or if " "atexit=1.") diff --git a/libsanitizer/asan/asan_globals.cpp b/libsanitizer/asan/asan_globals.cpp index e045c31cd1c..9d7dbc6f264 100644 --- a/libsanitizer/asan/asan_globals.cpp +++ b/libsanitizer/asan/asan_globals.cpp @@ -154,6 +154,23 @@ static void CheckODRViolationViaIndicator(const Global *g) { } } +// Check ODR violation for given global G by checking if it's already poisoned. +// We use this method in case compiler doesn't use private aliases for global +// variables. +static void CheckODRViolationViaPoisoning(const Global *g) { + if (__asan_region_is_poisoned(g->beg, g->size_with_redzone)) { + // This check may not be enough: if the first global is much larger + // the entire redzone of the second global may be within the first global. + for (ListOfGlobals *l = list_of_all_globals; l; l = l->next) { + if (g->beg == l->g->beg && + (flags()->detect_odr_violation >= 2 || g->size != l->g->size) && + !IsODRViolationSuppressed(g->name)) + ReportODRViolation(g, FindRegistrationSite(g), + l->g, FindRegistrationSite(l->g)); + } + } +} + // Clang provides two different ways for global variables protection: // it can poison the global itself or its private alias. In former // case we may poison same symbol multiple times, that can help us to @@ -199,6 +216,8 @@ static void RegisterGlobal(const Global *g) { // where two globals with the same name are defined in different modules. if (UseODRIndicator(g)) CheckODRViolationViaIndicator(g); + else + CheckODRViolationViaPoisoning(g); } if (CanPoisonMemory()) PoisonRedZones(*g); diff --git a/libsanitizer/asan/asan_interceptors.cpp b/libsanitizer/asan/asan_interceptors.cpp index 9db7db89fa1..d0a6dd48a74 100644 --- a/libsanitizer/asan/asan_interceptors.cpp +++ b/libsanitizer/asan/asan_interceptors.cpp @@ -23,25 +23,25 @@ #include "lsan/lsan_common.h" #include "sanitizer_common/sanitizer_libc.h" -// There is no general interception at all on Fuchsia and RTEMS. +// There is no general interception at all on Fuchsia. // Only the functions in asan_interceptors_memintrinsics.cpp are // really defined to replace libc functions. -#if !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +#if !SANITIZER_FUCHSIA -#if SANITIZER_POSIX -#include "sanitizer_common/sanitizer_posix.h" -#endif +# if SANITIZER_POSIX +# include "sanitizer_common/sanitizer_posix.h" +# endif -#if ASAN_INTERCEPT__UNWIND_RAISEEXCEPTION || \ - ASAN_INTERCEPT__SJLJ_UNWIND_RAISEEXCEPTION -#include -#endif +# if ASAN_INTERCEPT__UNWIND_RAISEEXCEPTION || \ + ASAN_INTERCEPT__SJLJ_UNWIND_RAISEEXCEPTION +# include +# endif -#if defined(__i386) && SANITIZER_LINUX -#define ASAN_PTHREAD_CREATE_VERSION "GLIBC_2.1" -#elif defined(__mips__) && SANITIZER_LINUX -#define ASAN_PTHREAD_CREATE_VERSION "GLIBC_2.2" -#endif +# if defined(__i386) && SANITIZER_LINUX +# define ASAN_PTHREAD_CREATE_VERSION "GLIBC_2.1" +# elif defined(__mips__) && SANITIZER_LINUX +# define ASAN_PTHREAD_CREATE_VERSION "GLIBC_2.2" +# endif namespace __asan { diff --git a/libsanitizer/asan/asan_interceptors.h b/libsanitizer/asan/asan_interceptors.h index 5bd5be6bac2..a9249dea45b 100644 --- a/libsanitizer/asan/asan_interceptors.h +++ b/libsanitizer/asan/asan_interceptors.h @@ -34,10 +34,10 @@ void InitializePlatformInterceptors(); } // namespace __asan -// There is no general interception at all on Fuchsia and RTEMS. +// There is no general interception at all on Fuchsia. // Only the functions in asan_interceptors_memintrinsics.h are // really defined to replace libc functions. -#if !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +#if !SANITIZER_FUCHSIA // Use macro to describe if specific function should be // intercepted on a given platform. @@ -81,12 +81,7 @@ void InitializePlatformInterceptors(); #if ASAN_HAS_EXCEPTIONS && !SANITIZER_WINDOWS && !SANITIZER_SOLARIS && \ !SANITIZER_NETBSD # define ASAN_INTERCEPT___CXA_THROW 1 -# if ! defined(ASAN_HAS_CXA_RETHROW_PRIMARY_EXCEPTION) \ - || ASAN_HAS_CXA_RETHROW_PRIMARY_EXCEPTION -# define ASAN_INTERCEPT___CXA_RETHROW_PRIMARY_EXCEPTION 1 -# else -# define ASAN_INTERCEPT___CXA_RETHROW_PRIMARY_EXCEPTION 0 -# endif +# define ASAN_INTERCEPT___CXA_RETHROW_PRIMARY_EXCEPTION 1 # if defined(_GLIBCXX_SJLJ_EXCEPTIONS) || (SANITIZER_IOS && defined(__arm__)) # define ASAN_INTERCEPT__UNWIND_SJLJ_RAISEEXCEPTION 1 # else diff --git a/libsanitizer/asan/asan_interceptors_memintrinsics.cpp b/libsanitizer/asan/asan_interceptors_memintrinsics.cpp index ccdd5159042..9c316bb9574 100644 --- a/libsanitizer/asan/asan_interceptors_memintrinsics.cpp +++ b/libsanitizer/asan/asan_interceptors_memintrinsics.cpp @@ -30,9 +30,9 @@ void *__asan_memmove(void *to, const void *from, uptr size) { ASAN_MEMMOVE_IMPL(nullptr, to, from, size); } -#if SANITIZER_FUCHSIA || SANITIZER_RTEMS +#if SANITIZER_FUCHSIA -// Fuchsia and RTEMS don't use sanitizer_common_interceptors.inc, but +// Fuchsia doesn't use sanitizer_common_interceptors.inc, but // the only things there it wants are these three. Just define them // as aliases here rather than repeating the contents. @@ -40,4 +40,4 @@ extern "C" decltype(__asan_memcpy) memcpy[[gnu::alias("__asan_memcpy")]]; extern "C" decltype(__asan_memmove) memmove[[gnu::alias("__asan_memmove")]]; extern "C" decltype(__asan_memset) memset[[gnu::alias("__asan_memset")]]; -#endif // SANITIZER_FUCHSIA || SANITIZER_RTEMS +#endif // SANITIZER_FUCHSIA diff --git a/libsanitizer/asan/asan_interface.inc b/libsanitizer/asan/asan_interface.inc index 94801043982..ea28fc8ae87 100644 --- a/libsanitizer/asan/asan_interface.inc +++ b/libsanitizer/asan/asan_interface.inc @@ -134,6 +134,17 @@ INTERFACE_FUNCTION(__asan_stack_malloc_7) INTERFACE_FUNCTION(__asan_stack_malloc_8) INTERFACE_FUNCTION(__asan_stack_malloc_9) INTERFACE_FUNCTION(__asan_stack_malloc_10) +INTERFACE_FUNCTION(__asan_stack_malloc_always_0) +INTERFACE_FUNCTION(__asan_stack_malloc_always_1) +INTERFACE_FUNCTION(__asan_stack_malloc_always_2) +INTERFACE_FUNCTION(__asan_stack_malloc_always_3) +INTERFACE_FUNCTION(__asan_stack_malloc_always_4) +INTERFACE_FUNCTION(__asan_stack_malloc_always_5) +INTERFACE_FUNCTION(__asan_stack_malloc_always_6) +INTERFACE_FUNCTION(__asan_stack_malloc_always_7) +INTERFACE_FUNCTION(__asan_stack_malloc_always_8) +INTERFACE_FUNCTION(__asan_stack_malloc_always_9) +INTERFACE_FUNCTION(__asan_stack_malloc_always_10) INTERFACE_FUNCTION(__asan_store1) INTERFACE_FUNCTION(__asan_store2) INTERFACE_FUNCTION(__asan_store4) diff --git a/libsanitizer/asan/asan_internal.h b/libsanitizer/asan/asan_internal.h index cfb54927c6c..ad3320304d0 100644 --- a/libsanitizer/asan/asan_internal.h +++ b/libsanitizer/asan/asan_internal.h @@ -35,11 +35,11 @@ // If set, values like allocator chunk size, as well as defaults for some flags // will be changed towards less memory overhead. #ifndef ASAN_LOW_MEMORY -# if SANITIZER_IOS || SANITIZER_ANDROID || SANITIZER_RTEMS -# define ASAN_LOW_MEMORY 1 -# else -# define ASAN_LOW_MEMORY 0 -# endif +# if SANITIZER_IOS || SANITIZER_ANDROID +# define ASAN_LOW_MEMORY 1 +# else +# define ASAN_LOW_MEMORY 0 +# endif #endif #ifndef ASAN_DYNAMIC @@ -77,7 +77,7 @@ void InitializeShadowMemory(); // asan_malloc_linux.cpp / asan_malloc_mac.cpp void ReplaceSystemMalloc(); -// asan_linux.cpp / asan_mac.cpp / asan_rtems.cpp / asan_win.cpp +// asan_linux.cpp / asan_mac.cpp / asan_win.cpp uptr FindDynamicShadowStart(); void *AsanDoesNotSupportStaticLinkage(); void AsanCheckDynamicRTPrereqs(); @@ -159,9 +159,6 @@ const int kAsanArrayCookieMagic = 0xac; const int kAsanIntraObjectRedzone = 0xbb; const int kAsanAllocaLeftMagic = 0xca; const int kAsanAllocaRightMagic = 0xcb; -// Used to populate the shadow gap for systems without memory -// protection there (i.e. Myriad). -const int kAsanShadowGap = 0xcc; static const uptr kCurrentStackFrameMagic = 0x41B58AB3; static const uptr kRetiredStackFrameMagic = 0x45E0360E; diff --git a/libsanitizer/asan/asan_malloc_linux.cpp b/libsanitizer/asan/asan_malloc_linux.cpp index 9c3f0a5338e..c6bec8551bc 100644 --- a/libsanitizer/asan/asan_malloc_linux.cpp +++ b/libsanitizer/asan/asan_malloc_linux.cpp @@ -15,23 +15,22 @@ #include "sanitizer_common/sanitizer_platform.h" #if SANITIZER_FREEBSD || SANITIZER_FUCHSIA || SANITIZER_LINUX || \ - SANITIZER_NETBSD || SANITIZER_RTEMS || SANITIZER_SOLARIS + SANITIZER_NETBSD || SANITIZER_SOLARIS -#include "sanitizer_common/sanitizer_allocator_checks.h" -#include "sanitizer_common/sanitizer_errno.h" -#include "sanitizer_common/sanitizer_tls_get_addr.h" -#include "asan_allocator.h" -#include "asan_interceptors.h" -#include "asan_internal.h" -#include "asan_malloc_local.h" -#include "asan_stack.h" +# include "asan_allocator.h" +# include "asan_interceptors.h" +# include "asan_internal.h" +# include "asan_stack.h" +# include "sanitizer_common/sanitizer_allocator_checks.h" +# include "sanitizer_common/sanitizer_errno.h" +# include "sanitizer_common/sanitizer_tls_get_addr.h" // ---------------------- Replacement functions ---------------- {{{1 using namespace __asan; static uptr allocated_for_dlsym; static uptr last_dlsym_alloc_size_in_words; -static const uptr kDlsymAllocPoolSize = SANITIZER_RTEMS ? 4096 : 1024; +static const uptr kDlsymAllocPoolSize = 1024; static uptr alloc_memory_for_dlsym[kDlsymAllocPoolSize]; static inline bool IsInDlsymAllocPool(const void *ptr) { @@ -82,27 +81,12 @@ static int PosixMemalignFromLocalPool(void **memptr, uptr alignment, return 0; } -#if SANITIZER_RTEMS -void* MemalignFromLocalPool(uptr alignment, uptr size) { - void *ptr = nullptr; - alignment = Max(alignment, kWordSize); - PosixMemalignFromLocalPool(&ptr, alignment, size); - return ptr; -} - -bool IsFromLocalPool(const void *ptr) { - return IsInDlsymAllocPool(ptr); -} -#endif - static inline bool MaybeInDlsym() { // Fuchsia doesn't use dlsym-based interceptors. return !SANITIZER_FUCHSIA && asan_init_is_running; } -static inline bool UseLocalPool() { - return EarlyMalloc() || MaybeInDlsym(); -} +static inline bool UseLocalPool() { return MaybeInDlsym(); } static void *ReallocFromLocalPool(void *ptr, uptr size) { const uptr offset = (uptr)ptr - (uptr)alloc_memory_for_dlsym; diff --git a/libsanitizer/asan/asan_malloc_local.h b/libsanitizer/asan/asan_malloc_local.h deleted file mode 100644 index e2c9be0379f..00000000000 --- a/libsanitizer/asan/asan_malloc_local.h +++ /dev/null @@ -1,52 +0,0 @@ -//===-- asan_malloc_local.h -------------------------------------*- C++ -*-===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// -// -// This file is a part of AddressSanitizer, an address sanity checker. -// -// Provide interfaces to check for and handle local pool memory allocation. -//===----------------------------------------------------------------------===// - -#ifndef ASAN_MALLOC_LOCAL_H -#define ASAN_MALLOC_LOCAL_H - -#include "sanitizer_common/sanitizer_platform.h" -#include "asan_internal.h" - -static inline bool EarlyMalloc() { - return SANITIZER_RTEMS && - (!__asan::asan_inited || __asan::asan_init_is_running); -} - -#if SANITIZER_RTEMS - -bool IsFromLocalPool(const void *ptr); -void *MemalignFromLocalPool(uptr alignment, uptr size); - -// On RTEMS, we use the local pool to handle memory allocation when the ASan -// run-time is not up. This macro is expanded in the context of the operator new -// implementation. -#define MAYBE_ALLOCATE_FROM_LOCAL_POOL(nothrow) \ - do { \ - if (UNLIKELY(EarlyMalloc())) { \ - void *res = MemalignFromLocalPool(SHADOW_GRANULARITY, size); \ - if (!nothrow) \ - CHECK(res); \ - return res; \ - } \ - } while (0) - -#define IS_FROM_LOCAL_POOL(ptr) UNLIKELY(IsFromLocalPool(ptr)) - -#else // SANITIZER_RTEMS - -#define MAYBE_ALLOCATE_FROM_LOCAL_POOL(nothrow) -#define IS_FROM_LOCAL_POOL(ptr) 0 - -#endif // SANITIZER_RTEMS - -#endif // ASAN_MALLOC_LOCAL_H diff --git a/libsanitizer/asan/asan_mapping.h b/libsanitizer/asan/asan_mapping.h index 2511e7043df..e5a7f2007ae 100644 --- a/libsanitizer/asan/asan_mapping.h +++ b/libsanitizer/asan/asan_mapping.h @@ -150,17 +150,11 @@ // || `[0x36000000, 0x39ffffff]` || ShadowGap || // || `[0x30000000, 0x35ffffff]` || LowShadow || // || `[0x00000000, 0x2fffffff]` || LowMem || -// -// Shadow mapping on Myriad2 (for shadow scale 5): -// || `[0x9ff80000, 0x9fffffff]` || ShadowGap || -// || `[0x9f000000, 0x9ff7ffff]` || LowShadow || -// || `[0x80000000, 0x9effffff]` || LowMem || -// || `[0x00000000, 0x7fffffff]` || Ignored || #if defined(ASAN_SHADOW_SCALE) static const u64 kDefaultShadowScale = ASAN_SHADOW_SCALE; #else -static const u64 kDefaultShadowScale = SANITIZER_MYRIAD2 ? 5 : 3; +static const u64 kDefaultShadowScale = 3; #endif static const u64 kDefaultShadowSentinel = ~(uptr)0; static const u64 kDefaultShadowOffset32 = 1ULL << 29; // 0x20000000 @@ -171,7 +165,7 @@ static const u64 kAArch64_ShadowOffset64 = 1ULL << 36; static const u64 kRiscv64_ShadowOffset64 = 0xd55550000; static const u64 kMIPS32_ShadowOffset32 = 0x0aaa0000; static const u64 kMIPS64_ShadowOffset64 = 1ULL << 37; -static const u64 kPPC64_ShadowOffset64 = 1ULL << 41; +static const u64 kPPC64_ShadowOffset64 = 1ULL << 44; static const u64 kSystemZ_ShadowOffset64 = 1ULL << 52; static const u64 kSPARC64_ShadowOffset64 = 1ULL << 43; // 0x80000000000 static const u64 kFreeBSD_ShadowOffset32 = 1ULL << 30; // 0x40000000 @@ -180,15 +174,6 @@ static const u64 kNetBSD_ShadowOffset32 = 1ULL << 30; // 0x40000000 static const u64 kNetBSD_ShadowOffset64 = 1ULL << 46; // 0x400000000000 static const u64 kWindowsShadowOffset32 = 3ULL << 28; // 0x30000000 -static const u64 kMyriadMemoryOffset32 = 0x80000000ULL; -static const u64 kMyriadMemorySize32 = 0x20000000ULL; -static const u64 kMyriadMemoryEnd32 = - kMyriadMemoryOffset32 + kMyriadMemorySize32 - 1; -static const u64 kMyriadShadowOffset32 = - (kMyriadMemoryOffset32 + kMyriadMemorySize32 - - (kMyriadMemorySize32 >> kDefaultShadowScale)); -static const u64 kMyriadCacheBitMask32 = 0x40000000ULL; - #define SHADOW_SCALE kDefaultShadowScale #if SANITIZER_FUCHSIA @@ -206,8 +191,6 @@ static const u64 kMyriadCacheBitMask32 = 0x40000000ULL; # define SHADOW_OFFSET kWindowsShadowOffset32 # elif SANITIZER_IOS # define SHADOW_OFFSET __asan_shadow_memory_dynamic_address -# elif SANITIZER_MYRIAD2 -# define SHADOW_OFFSET kMyriadShadowOffset32 # else # define SHADOW_OFFSET kDefaultShadowOffset32 # endif @@ -278,10 +261,8 @@ extern uptr kHighMemEnd, kMidMemBeg, kMidMemEnd; // Initialized in __asan_init. } // namespace __asan -#if SANITIZER_MYRIAD2 -#include "asan_mapping_myriad.h" -#elif defined(__sparc__) && SANITIZER_WORDSIZE == 64 -#include "asan_mapping_sparc64.h" +#if defined(__sparc__) && SANITIZER_WORDSIZE == 64 +# include "asan_mapping_sparc64.h" #else #define MEM_TO_SHADOW(mem) (((mem) >> SHADOW_SCALE) + (SHADOW_OFFSET)) @@ -363,7 +344,7 @@ static inline bool AddrIsInShadowGap(uptr a) { } // namespace __asan -#endif // SANITIZER_MYRIAD2 +#endif namespace __asan { @@ -393,8 +374,6 @@ static inline bool AddrIsAlignedByGranularity(uptr a) { static inline bool AddressIsPoisoned(uptr a) { PROFILE_ASAN_MAPPING(); - if (SANITIZER_MYRIAD2 && !AddrIsInMem(a) && !AddrIsInShadow(a)) - return false; const uptr kAccessSize = 1; u8 *shadow_address = (u8*)MEM_TO_SHADOW(a); s8 shadow_value = *shadow_address; diff --git a/libsanitizer/asan/asan_mapping_myriad.h b/libsanitizer/asan/asan_mapping_myriad.h deleted file mode 100644 index 6969e3a4931..00000000000 --- a/libsanitizer/asan/asan_mapping_myriad.h +++ /dev/null @@ -1,85 +0,0 @@ -//===-- asan_mapping_myriad.h -----------------------------------*- C++ -*-===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// -// -// This file is a part of AddressSanitizer, an address sanity checker. -// -// Myriad-specific definitions for ASan memory mapping. -//===----------------------------------------------------------------------===// -#ifndef ASAN_MAPPING_MYRIAD_H -#define ASAN_MAPPING_MYRIAD_H - -#define RAW_ADDR(mem) ((mem) & ~kMyriadCacheBitMask32) -#define MEM_TO_SHADOW(mem) \ - (((RAW_ADDR(mem) - kLowMemBeg) >> SHADOW_SCALE) + (SHADOW_OFFSET)) - -#define kLowMemBeg kMyriadMemoryOffset32 -#define kLowMemEnd (SHADOW_OFFSET - 1) - -#define kLowShadowBeg SHADOW_OFFSET -#define kLowShadowEnd MEM_TO_SHADOW(kLowMemEnd) - -#define kHighMemBeg 0 - -#define kHighShadowBeg 0 -#define kHighShadowEnd 0 - -#define kMidShadowBeg 0 -#define kMidShadowEnd 0 - -#define kShadowGapBeg (kLowShadowEnd + 1) -#define kShadowGapEnd kMyriadMemoryEnd32 - -#define kShadowGap2Beg 0 -#define kShadowGap2End 0 - -#define kShadowGap3Beg 0 -#define kShadowGap3End 0 - -namespace __asan { - -static inline bool AddrIsInLowMem(uptr a) { - PROFILE_ASAN_MAPPING(); - a = RAW_ADDR(a); - return a >= kLowMemBeg && a <= kLowMemEnd; -} - -static inline bool AddrIsInLowShadow(uptr a) { - PROFILE_ASAN_MAPPING(); - a = RAW_ADDR(a); - return a >= kLowShadowBeg && a <= kLowShadowEnd; -} - -static inline bool AddrIsInMidMem(uptr a) { - PROFILE_ASAN_MAPPING(); - return false; -} - -static inline bool AddrIsInMidShadow(uptr a) { - PROFILE_ASAN_MAPPING(); - return false; -} - -static inline bool AddrIsInHighMem(uptr a) { - PROFILE_ASAN_MAPPING(); - return false; -} - -static inline bool AddrIsInHighShadow(uptr a) { - PROFILE_ASAN_MAPPING(); - return false; -} - -static inline bool AddrIsInShadowGap(uptr a) { - PROFILE_ASAN_MAPPING(); - a = RAW_ADDR(a); - return a >= kShadowGapBeg && a <= kShadowGapEnd; -} - -} // namespace __asan - -#endif // ASAN_MAPPING_MYRIAD_H diff --git a/libsanitizer/asan/asan_new_delete.cpp b/libsanitizer/asan/asan_new_delete.cpp index 92a8648452b..da446072de1 100644 --- a/libsanitizer/asan/asan_new_delete.cpp +++ b/libsanitizer/asan/asan_new_delete.cpp @@ -11,16 +11,14 @@ // Interceptors for operators new and delete. //===----------------------------------------------------------------------===// +#include + #include "asan_allocator.h" #include "asan_internal.h" -#include "asan_malloc_local.h" #include "asan_report.h" #include "asan_stack.h" - #include "interception/interception.h" -#include - // C++ operators can't have dllexport attributes on Windows. We export them // anyway by passing extra -export flags to the linker, which is exactly that // dllexport would normally do. We need to export them in order to make the @@ -72,14 +70,12 @@ enum class align_val_t: size_t {}; // For local pool allocation, align to SHADOW_GRANULARITY to match asan // allocator behavior. #define OPERATOR_NEW_BODY(type, nothrow) \ - MAYBE_ALLOCATE_FROM_LOCAL_POOL(nothrow); \ GET_STACK_TRACE_MALLOC; \ void *res = asan_memalign(0, size, &stack, type); \ if (!nothrow && UNLIKELY(!res)) \ ReportOutOfMemory(size, &stack); \ return res; #define OPERATOR_NEW_BODY_ALIGN(type, nothrow) \ - MAYBE_ALLOCATE_FROM_LOCAL_POOL(nothrow); \ GET_STACK_TRACE_MALLOC; \ void *res = asan_memalign((uptr)align, size, &stack, type); \ if (!nothrow && UNLIKELY(!res)) \ @@ -135,23 +131,19 @@ INTERCEPTOR(void *, _ZnamRKSt9nothrow_t, size_t size, std::nothrow_t const&) { #endif // !SANITIZER_MAC #define OPERATOR_DELETE_BODY(type) \ - if (IS_FROM_LOCAL_POOL(ptr)) return;\ - GET_STACK_TRACE_FREE;\ + GET_STACK_TRACE_FREE; \ asan_delete(ptr, 0, 0, &stack, type); #define OPERATOR_DELETE_BODY_SIZE(type) \ - if (IS_FROM_LOCAL_POOL(ptr)) return;\ - GET_STACK_TRACE_FREE;\ + GET_STACK_TRACE_FREE; \ asan_delete(ptr, size, 0, &stack, type); #define OPERATOR_DELETE_BODY_ALIGN(type) \ - if (IS_FROM_LOCAL_POOL(ptr)) return;\ - GET_STACK_TRACE_FREE;\ + GET_STACK_TRACE_FREE; \ asan_delete(ptr, 0, static_cast(align), &stack, type); #define OPERATOR_DELETE_BODY_SIZE_ALIGN(type) \ - if (IS_FROM_LOCAL_POOL(ptr)) return;\ - GET_STACK_TRACE_FREE;\ + GET_STACK_TRACE_FREE; \ asan_delete(ptr, size, static_cast(align), &stack, type); #if !SANITIZER_MAC diff --git a/libsanitizer/asan/asan_poisoning.cpp b/libsanitizer/asan/asan_poisoning.cpp index fa149ecfde6..5f215fe0f9b 100644 --- a/libsanitizer/asan/asan_poisoning.cpp +++ b/libsanitizer/asan/asan_poisoning.cpp @@ -173,17 +173,13 @@ int __asan_address_is_poisoned(void const volatile *addr) { } uptr __asan_region_is_poisoned(uptr beg, uptr size) { - if (!size) return 0; + if (!size) + return 0; uptr end = beg + size; - if (SANITIZER_MYRIAD2) { - // On Myriad, address not in DRAM range need to be treated as - // unpoisoned. - if (!AddrIsInMem(beg) && !AddrIsInShadow(beg)) return 0; - if (!AddrIsInMem(end) && !AddrIsInShadow(end)) return 0; - } else { - if (!AddrIsInMem(beg)) return beg; - if (!AddrIsInMem(end)) return end; - } + if (!AddrIsInMem(beg)) + return beg; + if (!AddrIsInMem(end)) + return end; CHECK_LT(beg, end); uptr aligned_b = RoundUpTo(beg, SHADOW_GRANULARITY); uptr aligned_e = RoundDownTo(end, SHADOW_GRANULARITY); @@ -192,8 +188,7 @@ uptr __asan_region_is_poisoned(uptr beg, uptr size) { // First check the first and the last application bytes, // then check the SHADOW_GRANULARITY-aligned region by calling // mem_is_zero on the corresponding shadow. - if (!__asan::AddressIsPoisoned(beg) && - !__asan::AddressIsPoisoned(end - 1) && + if (!__asan::AddressIsPoisoned(beg) && !__asan::AddressIsPoisoned(end - 1) && (shadow_end <= shadow_beg || __sanitizer::mem_is_zero((const char *)shadow_beg, shadow_end - shadow_beg))) diff --git a/libsanitizer/asan/asan_poisoning.h b/libsanitizer/asan/asan_poisoning.h index 62dd9bd0edd..3d536f2d309 100644 --- a/libsanitizer/asan/asan_poisoning.h +++ b/libsanitizer/asan/asan_poisoning.h @@ -51,9 +51,6 @@ ALWAYS_INLINE void FastPoisonShadow(uptr aligned_beg, uptr aligned_size, // probably provide higher-level interface for these operations. // For now, just memset on Windows. if (value || SANITIZER_WINDOWS == 1 || - // RTEMS doesn't have have pages, let alone a fast way to zero - // them, so default to memset. - SANITIZER_RTEMS == 1 || shadow_end - shadow_beg < common_flags()->clear_shadow_mmap_threshold) { REAL(memset)((void*)shadow_beg, value, shadow_end - shadow_beg); } else { diff --git a/libsanitizer/asan/asan_rtems.cpp b/libsanitizer/asan/asan_rtems.cpp deleted file mode 100644 index ea0b4ad9db6..00000000000 --- a/libsanitizer/asan/asan_rtems.cpp +++ /dev/null @@ -1,266 +0,0 @@ -//===-- asan_rtems.cpp ----------------------------------------------------===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// -// -// This file is a part of AddressSanitizer, an address sanity checker. -// -// RTEMS-specific details. -//===----------------------------------------------------------------------===// - -#include "sanitizer_common/sanitizer_rtems.h" -#if SANITIZER_RTEMS - -#include "asan_internal.h" -#include "asan_interceptors.h" -#include "asan_mapping.h" -#include "asan_poisoning.h" -#include "asan_report.h" -#include "asan_stack.h" -#include "sanitizer_common/sanitizer_common.h" -#include "sanitizer_common/sanitizer_libc.h" - -#include -#include - -namespace __asan { - -static void ResetShadowMemory() { - uptr shadow_start = SHADOW_OFFSET; - uptr shadow_end = MEM_TO_SHADOW(kMyriadMemoryEnd32); - uptr gap_start = MEM_TO_SHADOW(shadow_start); - uptr gap_end = MEM_TO_SHADOW(shadow_end); - - REAL(memset)((void *)shadow_start, 0, shadow_end - shadow_start); - REAL(memset)((void *)gap_start, kAsanShadowGap, gap_end - gap_start); -} - -void InitializeShadowMemory() { - kHighMemEnd = 0; - kMidMemBeg = 0; - kMidMemEnd = 0; - - ResetShadowMemory(); -} - -void AsanApplyToGlobals(globals_op_fptr op, const void *needle) { - UNIMPLEMENTED(); -} - -void FlushUnneededASanShadowMemory(uptr p, uptr size) { - // Since asan's mapping is compacting, the shadow chunk may be - // not page-aligned, so we only flush the page-aligned portion. - ReleaseMemoryPagesToOS(MemToShadow(p), MemToShadow(p + size)); -} - -void AsanCheckDynamicRTPrereqs() {} -void AsanCheckIncompatibleRT() {} -void InitializeAsanInterceptors() {} -void InitializePlatformInterceptors() {} -void InitializePlatformExceptionHandlers() {} - -// RTEMS only support static linking; it sufficies to return with no -// error. -void *AsanDoesNotSupportStaticLinkage() { return nullptr; } - -void AsanOnDeadlySignal(int signo, void *siginfo, void *context) { - UNIMPLEMENTED(); -} - -bool PlatformUnpoisonStacks() { return false; } - -void EarlyInit() { - // Provide early initialization of shadow memory so that - // instrumented code running before full initialzation will not - // report spurious errors. - ResetShadowMemory(); -} - -// We can use a plain thread_local variable for TSD. -static thread_local void *per_thread; - -void *AsanTSDGet() { return per_thread; } - -void AsanTSDSet(void *tsd) { per_thread = tsd; } - -// There's no initialization needed, and the passed-in destructor -// will never be called. Instead, our own thread destruction hook -// (below) will call AsanThread::TSDDtor directly. -void AsanTSDInit(void (*destructor)(void *tsd)) { - DCHECK(destructor == &PlatformTSDDtor); -} - -void PlatformTSDDtor(void *tsd) { UNREACHABLE(__func__); } - -// -// Thread registration. We provide an API similar to the Fushia port. -// - -struct AsanThread::InitOptions { - uptr stack_bottom, stack_size, tls_bottom, tls_size; -}; - -// Shared setup between thread creation and startup for the initial thread. -static AsanThread *CreateAsanThread(StackTrace *stack, u32 parent_tid, - uptr user_id, bool detached, - uptr stack_bottom, uptr stack_size, - uptr tls_bottom, uptr tls_size) { - // In lieu of AsanThread::Create. - AsanThread *thread = (AsanThread *)MmapOrDie(sizeof(AsanThread), __func__); - AsanThreadContext::CreateThreadContextArgs args = {thread, stack}; - asanThreadRegistry().CreateThread(user_id, detached, parent_tid, &args); - - // On other systems, AsanThread::Init() is called from the new - // thread itself. But on RTEMS we already know the stack address - // range beforehand, so we can do most of the setup right now. - const AsanThread::InitOptions options = {stack_bottom, stack_size, - tls_bottom, tls_size}; - thread->Init(&options); - return thread; -} - -// This gets the same arguments passed to Init by CreateAsanThread, above. -// We're in the creator thread before the new thread is actually started, but -// its stack and tls address range are already known. -void AsanThread::SetThreadStackAndTls(const AsanThread::InitOptions *options) { - DCHECK_NE(GetCurrentThread(), this); - DCHECK_NE(GetCurrentThread(), nullptr); - CHECK_NE(options->stack_bottom, 0); - CHECK_NE(options->stack_size, 0); - stack_bottom_ = options->stack_bottom; - stack_top_ = options->stack_bottom + options->stack_size; - tls_begin_ = options->tls_bottom; - tls_end_ = options->tls_bottom + options->tls_size; -} - -// Called by __asan::AsanInitInternal (asan_rtl.c). Unlike other ports, the -// main thread on RTEMS does not require special treatment; its AsanThread is -// already created by the provided hooks. This function simply looks up and -// returns the created thread. -AsanThread *CreateMainThread() { - return GetThreadContextByTidLocked(0)->thread; -} - -// This is called before each thread creation is attempted. So, in -// its first call, the calling thread is the initial and sole thread. -static void *BeforeThreadCreateHook(uptr user_id, bool detached, - uptr stack_bottom, uptr stack_size, - uptr tls_bottom, uptr tls_size) { - EnsureMainThreadIDIsCorrect(); - // Strict init-order checking is thread-hostile. - if (flags()->strict_init_order) StopInitOrderChecking(); - - GET_STACK_TRACE_THREAD; - u32 parent_tid = GetCurrentTidOrInvalid(); - - return CreateAsanThread(&stack, parent_tid, user_id, detached, - stack_bottom, stack_size, tls_bottom, tls_size); -} - -// This is called after creating a new thread (in the creating thread), -// with the pointer returned by BeforeThreadCreateHook (above). -static void ThreadCreateHook(void *hook, bool aborted) { - AsanThread *thread = static_cast(hook); - if (!aborted) { - // The thread was created successfully. - // ThreadStartHook is already running in the new thread. - } else { - // The thread wasn't created after all. - // Clean up everything we set up in BeforeThreadCreateHook. - asanThreadRegistry().FinishThread(thread->tid()); - UnmapOrDie(thread, sizeof(AsanThread)); - } -} - -// This is called (1) in the newly-created thread before it runs anything else, -// with the pointer returned by BeforeThreadCreateHook (above). (2) before a -// thread restart. -static void ThreadStartHook(void *hook, uptr os_id) { - if (!hook) - return; - - AsanThread *thread = static_cast(hook); - SetCurrentThread(thread); - - ThreadStatus status = - asanThreadRegistry().GetThreadLocked(thread->tid())->status; - DCHECK(status == ThreadStatusCreated || status == ThreadStatusRunning); - // Determine whether we are starting or restarting the thread. - if (status == ThreadStatusCreated) { - // In lieu of AsanThread::ThreadStart. - asanThreadRegistry().StartThread(thread->tid(), os_id, ThreadType::Regular, - nullptr); - } else { - // In a thread restart, a thread may resume execution at an - // arbitrary function entry point, with its stack and TLS state - // reset. We unpoison the stack in that case. - PoisonShadow(thread->stack_bottom(), thread->stack_size(), 0); - } -} - -// Each thread runs this just before it exits, -// with the pointer returned by BeforeThreadCreateHook (above). -// All per-thread destructors have already been called. -static void ThreadExitHook(void *hook, uptr os_id) { - AsanThread *thread = static_cast(hook); - if (thread) - AsanThread::TSDDtor(thread->context()); -} - -static void HandleExit() { - // Disable ASan by setting it to uninitialized. Also reset the - // shadow memory to avoid reporting errors after the run-time has - // been desroyed. - if (asan_inited) { - asan_inited = false; - ResetShadowMemory(); - } -} - -bool HandleDlopenInit() { - // Not supported on this platform. - static_assert(!SANITIZER_SUPPORTS_INIT_FOR_DLOPEN, - "Expected SANITIZER_SUPPORTS_INIT_FOR_DLOPEN to be false"); - return false; -} -} // namespace __asan - -// These are declared (in extern "C") by . -// The system runtime will call our definitions directly. - -extern "C" { -void __sanitizer_early_init() { - __asan::EarlyInit(); -} - -void *__sanitizer_before_thread_create_hook(uptr thread, bool detached, - const char *name, - void *stack_base, size_t stack_size, - void *tls_base, size_t tls_size) { - return __asan::BeforeThreadCreateHook( - thread, detached, - reinterpret_cast(stack_base), stack_size, - reinterpret_cast(tls_base), tls_size); -} - -void __sanitizer_thread_create_hook(void *handle, uptr thread, int status) { - __asan::ThreadCreateHook(handle, status != 0); -} - -void __sanitizer_thread_start_hook(void *handle, uptr self) { - __asan::ThreadStartHook(handle, self); -} - -void __sanitizer_thread_exit_hook(void *handle, uptr self) { - __asan::ThreadExitHook(handle, self); -} - -void __sanitizer_exit() { - __asan::HandleExit(); -} -} // "C" - -#endif // SANITIZER_RTEMS diff --git a/libsanitizer/asan/asan_rtl.cpp b/libsanitizer/asan/asan_rtl.cpp index e715d774228..e06a1113f4e 100644 --- a/libsanitizer/asan/asan_rtl.cpp +++ b/libsanitizer/asan/asan_rtl.cpp @@ -13,6 +13,7 @@ #include "asan_activation.h" #include "asan_allocator.h" +#include "asan_fake_stack.h" #include "asan_interceptors.h" #include "asan_interface_internal.h" #include "asan_internal.h" @@ -23,11 +24,11 @@ #include "asan_stats.h" #include "asan_suppressions.h" #include "asan_thread.h" +#include "lsan/lsan_common.h" #include "sanitizer_common/sanitizer_atomic.h" #include "sanitizer_common/sanitizer_flags.h" #include "sanitizer_common/sanitizer_libc.h" #include "sanitizer_common/sanitizer_symbolizer.h" -#include "lsan/lsan_common.h" #include "ubsan/ubsan_init.h" #include "ubsan/ubsan_platform.h" @@ -137,24 +138,21 @@ ASAN_REPORT_ERROR_N(load, false) ASAN_REPORT_ERROR_N(store, true) #define ASAN_MEMORY_ACCESS_CALLBACK_BODY(type, is_write, size, exp_arg, fatal) \ - if (SANITIZER_MYRIAD2 && !AddrIsInMem(addr) && !AddrIsInShadow(addr)) \ - return; \ - uptr sp = MEM_TO_SHADOW(addr); \ - uptr s = size <= SHADOW_GRANULARITY ? *reinterpret_cast(sp) \ - : *reinterpret_cast(sp); \ - if (UNLIKELY(s)) { \ - if (UNLIKELY(size >= SHADOW_GRANULARITY || \ - ((s8)((addr & (SHADOW_GRANULARITY - 1)) + size - 1)) >= \ - (s8)s)) { \ - if (__asan_test_only_reported_buggy_pointer) { \ - *__asan_test_only_reported_buggy_pointer = addr; \ - } else { \ - GET_CALLER_PC_BP_SP; \ - ReportGenericError(pc, bp, sp, addr, is_write, size, exp_arg, \ - fatal); \ - } \ + uptr sp = MEM_TO_SHADOW(addr); \ + uptr s = size <= SHADOW_GRANULARITY ? *reinterpret_cast(sp) \ + : *reinterpret_cast(sp); \ + if (UNLIKELY(s)) { \ + if (UNLIKELY(size >= SHADOW_GRANULARITY || \ + ((s8)((addr & (SHADOW_GRANULARITY - 1)) + size - 1)) >= \ + (s8)s)) { \ + if (__asan_test_only_reported_buggy_pointer) { \ + *__asan_test_only_reported_buggy_pointer = addr; \ + } else { \ + GET_CALLER_PC_BP_SP; \ + ReportGenericError(pc, bp, sp, addr, is_write, size, exp_arg, fatal); \ } \ - } + } \ + } #define ASAN_MEMORY_ACCESS_CALLBACK(type, is_write, size) \ extern "C" NOINLINE INTERFACE_ATTRIBUTE \ @@ -305,7 +303,6 @@ static void asan_atexit() { } static void InitializeHighMemEnd() { -#if !SANITIZER_MYRIAD2 #if !ASAN_FIXED_MAPPING kHighMemEnd = GetMaxUserVirtualAddress(); // Increase kHighMemEnd to make sure it's properly @@ -313,7 +310,6 @@ static void InitializeHighMemEnd() { kHighMemEnd |= (GetMmapGranularity() << SHADOW_SCALE) - 1; #endif // !ASAN_FIXED_MAPPING CHECK_EQ((kHighMemBeg % GetMmapGranularity()), 0); -#endif // !SANITIZER_MYRIAD2 } void PrintAddressSpaceLayout() { @@ -569,9 +565,6 @@ static void UnpoisonDefaultStack() { const uptr page_size = GetPageSizeCached(); top = curr_thread->stack_top(); bottom = ((uptr)&local_stack - page_size) & ~(page_size - 1); - } else if (SANITIZER_RTEMS) { - // Give up On RTEMS. - return; } else { CHECK(!SANITIZER_FUCHSIA); // If we haven't seen this thread, try asking the OS for stack bounds. @@ -586,8 +579,12 @@ static void UnpoisonDefaultStack() { static void UnpoisonFakeStack() { AsanThread *curr_thread = GetCurrentThread(); - if (curr_thread && curr_thread->has_fake_stack()) - curr_thread->fake_stack()->HandleNoReturn(); + if (!curr_thread) + return; + FakeStack *stack = curr_thread->get_fake_stack(); + if (!stack) + return; + stack->HandleNoReturn(); } } // namespace __asan diff --git a/libsanitizer/asan/asan_shadow_setup.cpp b/libsanitizer/asan/asan_shadow_setup.cpp index 2ead4425add..6e6260d3413 100644 --- a/libsanitizer/asan/asan_shadow_setup.cpp +++ b/libsanitizer/asan/asan_shadow_setup.cpp @@ -13,12 +13,11 @@ #include "sanitizer_common/sanitizer_platform.h" -// asan_fuchsia.cpp and asan_rtems.cpp have their own -// InitializeShadowMemory implementation. -#if !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +// asan_fuchsia.cpp has their own InitializeShadowMemory implementation. +#if !SANITIZER_FUCHSIA -#include "asan_internal.h" -#include "asan_mapping.h" +# include "asan_internal.h" +# include "asan_mapping.h" namespace __asan { @@ -123,4 +122,4 @@ void InitializeShadowMemory() { } // namespace __asan -#endif // !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +#endif // !SANITIZER_FUCHSIA diff --git a/libsanitizer/asan/asan_stack.cpp b/libsanitizer/asan/asan_stack.cpp index b7f4e6aeeab..048295d6928 100644 --- a/libsanitizer/asan/asan_stack.cpp +++ b/libsanitizer/asan/asan_stack.cpp @@ -74,7 +74,8 @@ void __sanitizer::BufferedStackTrace::UnwindImpl( if (SANITIZER_MIPS && t && !IsValidFrame(bp, t->stack_top(), t->stack_bottom())) return; - Unwind(max_depth, pc, bp, context, 0, 0, false); + Unwind(max_depth, pc, bp, context, t ? t->stack_top() : 0, + t ? t->stack_bottom() : 0, false); } // ------------------ Interface -------------- {{{1 diff --git a/libsanitizer/asan/asan_thread.cpp b/libsanitizer/asan/asan_thread.cpp index 9c3c86f5735..35d4467e7b5 100644 --- a/libsanitizer/asan/asan_thread.cpp +++ b/libsanitizer/asan/asan_thread.cpp @@ -60,8 +60,8 @@ ThreadRegistry &asanThreadRegistry() { // in TSD and can't reliably tell when no more TSD destructors will // be called. It would be wrong to reuse AsanThreadContext for another // thread before all TSD destructors will be called for it. - asan_thread_registry = new(thread_registry_placeholder) ThreadRegistry( - GetAsanThreadContext, kMaxNumberOfThreads, kMaxNumberOfThreads); + asan_thread_registry = + new (thread_registry_placeholder) ThreadRegistry(GetAsanThreadContext); initialized = true; } return *asan_thread_registry; @@ -257,10 +257,9 @@ void AsanThread::Init(const InitOptions *options) { &local); } -// Fuchsia and RTEMS don't use ThreadStart. -// asan_fuchsia.c/asan_rtems.c define CreateMainThread and -// SetThreadStackAndTls. -#if !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +// Fuchsia doesn't use ThreadStart. +// asan_fuchsia.c definies CreateMainThread and SetThreadStackAndTls. +#if !SANITIZER_FUCHSIA thread_return_t AsanThread::ThreadStart(tid_t os_id) { Init(); @@ -317,7 +316,7 @@ void AsanThread::SetThreadStackAndTls(const InitOptions *options) { } } -#endif // !SANITIZER_FUCHSIA && !SANITIZER_RTEMS +#endif // !SANITIZER_FUCHSIA void AsanThread::ClearShadowForThreadStackAndTLS() { if (stack_top_ != stack_bottom_) @@ -339,8 +338,8 @@ bool AsanThread::GetStackFrameAccessByAddr(uptr addr, uptr bottom = 0; if (AddrIsInStack(addr)) { bottom = stack_bottom(); - } else if (has_fake_stack()) { - bottom = fake_stack()->AddrIsInFakeStack(addr); + } else if (FakeStack *fake_stack = get_fake_stack()) { + bottom = fake_stack->AddrIsInFakeStack(addr); CHECK(bottom); access->offset = addr - bottom; access->frame_pc = ((uptr*)bottom)[2]; @@ -380,8 +379,8 @@ uptr AsanThread::GetStackVariableShadowStart(uptr addr) { uptr bottom = 0; if (AddrIsInStack(addr)) { bottom = stack_bottom(); - } else if (has_fake_stack()) { - bottom = fake_stack()->AddrIsInFakeStack(addr); + } else if (FakeStack *fake_stack = get_fake_stack()) { + bottom = fake_stack->AddrIsInFakeStack(addr); if (bottom == 0) { return 0; } @@ -409,19 +408,19 @@ bool AsanThread::AddrIsInStack(uptr addr) { static bool ThreadStackContainsAddress(ThreadContextBase *tctx_base, void *addr) { - AsanThreadContext *tctx = static_cast(tctx_base); + AsanThreadContext *tctx = static_cast(tctx_base); AsanThread *t = tctx->thread; - if (!t) return false; - if (t->AddrIsInStack((uptr)addr)) return true; - if (t->has_fake_stack() && t->fake_stack()->AddrIsInFakeStack((uptr)addr)) + if (!t) + return false; + if (t->AddrIsInStack((uptr)addr)) return true; - return false; + FakeStack *fake_stack = t->get_fake_stack(); + if (!fake_stack) + return false; + return fake_stack->AddrIsInFakeStack((uptr)addr); } AsanThread *GetCurrentThread() { - if (SANITIZER_RTEMS && !asan_inited) - return nullptr; - AsanThreadContext *context = reinterpret_cast(AsanTSDGet()); if (!context) { @@ -503,8 +502,12 @@ void GetAllThreadAllocatorCachesLocked(InternalMmapVector *caches) {} void ForEachExtraStackRange(tid_t os_id, RangeIteratorCallback callback, void *arg) { __asan::AsanThread *t = __asan::GetAsanThreadByOsIDLocked(os_id); - if (t && t->has_fake_stack()) - t->fake_stack()->ForEachFakeFrame(callback, arg); + if (!t) + return; + __asan::FakeStack *fake_stack = t->get_fake_stack(); + if (!fake_stack) + return; + fake_stack->ForEachFakeFrame(callback, arg); } void LockThreadRegistry() { diff --git a/libsanitizer/asan/asan_thread.h b/libsanitizer/asan/asan_thread.h index 200069ce0dd..801a3960ec6 100644 --- a/libsanitizer/asan/asan_thread.h +++ b/libsanitizer/asan/asan_thread.h @@ -28,8 +28,6 @@ struct DTLS; namespace __asan { -const u32 kMaxNumberOfThreads = (1 << 22); // 4M - class AsanThread; // These objects are created for every thread and are never deleted, @@ -104,17 +102,18 @@ class AsanThread { void FinishSwitchFiber(FakeStack *fake_stack_save, uptr *bottom_old, uptr *size_old); - bool has_fake_stack() { - return !atomic_load(&stack_switching_, memory_order_relaxed) && - (reinterpret_cast(fake_stack_) > 1); + FakeStack *get_fake_stack() { + if (atomic_load(&stack_switching_, memory_order_relaxed)) + return nullptr; + if (reinterpret_cast(fake_stack_) <= 1) + return nullptr; + return fake_stack_; } - FakeStack *fake_stack() { - if (!__asan_option_detect_stack_use_after_return) - return nullptr; + FakeStack *get_or_create_fake_stack() { if (atomic_load(&stack_switching_, memory_order_relaxed)) return nullptr; - if (!has_fake_stack()) + if (reinterpret_cast(fake_stack_) <= 1) return AsyncSignalSafeLazyInitFakeStack(); return fake_stack_; } diff --git a/libsanitizer/hwasan/Makefile.am b/libsanitizer/hwasan/Makefile.am index 2226f68ab29..5e3a0f1b0a1 100644 --- a/libsanitizer/hwasan/Makefile.am +++ b/libsanitizer/hwasan/Makefile.am @@ -13,11 +13,13 @@ ACLOCAL_AMFLAGS = -I $(top_srcdir) -I $(top_srcdir)/config toolexeclib_LTLIBRARIES = libhwasan.la hwasan_files = \ + hwasan_allocation_functions.cpp \ hwasan_allocator.cpp \ hwasan.cpp \ hwasan_dynamic_shadow.cpp \ hwasan_exceptions.cpp \ hwasan_flags.inc \ + hwasan_fuchsia.cpp \ hwasan_globals.cpp \ hwasan_interceptors.cpp \ hwasan_interceptors_vfork.S \ diff --git a/libsanitizer/hwasan/Makefile.in b/libsanitizer/hwasan/Makefile.in index 542af8f193e..22c5266a120 100644 --- a/libsanitizer/hwasan/Makefile.in +++ b/libsanitizer/hwasan/Makefile.in @@ -146,8 +146,9 @@ am__DEPENDENCIES_1 = libhwasan_la_DEPENDENCIES = \ $(top_builddir)/sanitizer_common/libsanitizer_common.la \ $(am__append_1) $(am__append_2) $(am__DEPENDENCIES_1) -am__objects_1 = hwasan_allocator.lo hwasan.lo hwasan_dynamic_shadow.lo \ - hwasan_exceptions.lo hwasan_globals.lo hwasan_interceptors.lo \ +am__objects_1 = hwasan_allocation_functions.lo hwasan_allocator.lo \ + hwasan.lo hwasan_dynamic_shadow.lo hwasan_exceptions.lo \ + hwasan_fuchsia.lo hwasan_globals.lo hwasan_interceptors.lo \ hwasan_interceptors_vfork.lo hwasan_linux.lo \ hwasan_memintrinsics.lo hwasan_new_delete.lo \ hwasan_poisoning.lo hwasan_report.lo hwasan_setjmp.lo \ @@ -411,11 +412,13 @@ AM_CXXFLAGS = -Wall -W -Wno-unused-parameter -Wwrite-strings -pedantic \ ACLOCAL_AMFLAGS = -I $(top_srcdir) -I $(top_srcdir)/config toolexeclib_LTLIBRARIES = libhwasan.la hwasan_files = \ + hwasan_allocation_functions.cpp \ hwasan_allocator.cpp \ hwasan.cpp \ hwasan_dynamic_shadow.cpp \ hwasan_exceptions.cpp \ hwasan_flags.inc \ + hwasan_fuchsia.cpp \ hwasan_globals.cpp \ hwasan_interceptors.cpp \ hwasan_interceptors_vfork.S \ @@ -554,9 +557,11 @@ distclean-compile: -rm -f *.tab.c @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan.Plo@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_allocation_functions.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_allocator.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_dynamic_shadow.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_exceptions.Plo@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_fuchsia.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_globals.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_interceptors.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hwasan_interceptors_vfork.Plo@am__quote@ diff --git a/libsanitizer/hwasan/hwasan.cpp b/libsanitizer/hwasan/hwasan.cpp index 8d6c25261b8..cbe0dee66dc 100644 --- a/libsanitizer/hwasan/hwasan.cpp +++ b/libsanitizer/hwasan/hwasan.cpp @@ -50,6 +50,11 @@ bool hwasan_init_is_running; int hwasan_report_count = 0; +uptr kLowShadowStart; +uptr kLowShadowEnd; +uptr kHighShadowStart; +uptr kHighShadowEnd; + void Flags::SetDefaults() { #define HWASAN_FLAG(Type, Name, DefaultValue, Description) Name = DefaultValue; #include "hwasan_flags.inc" @@ -177,6 +182,65 @@ void UpdateMemoryUsage() { void UpdateMemoryUsage() {} #endif +void HwasanAtExit() { + if (common_flags()->print_module_map) + DumpProcessMap(); + if (flags()->print_stats && (flags()->atexit || hwasan_report_count > 0)) + ReportStats(); + if (hwasan_report_count > 0) { + // ReportAtExitStatistics(); + if (common_flags()->exitcode) + internal__exit(common_flags()->exitcode); + } +} + +void HandleTagMismatch(AccessInfo ai, uptr pc, uptr frame, void *uc, + uptr *registers_frame) { + InternalMmapVector stack_buffer(1); + BufferedStackTrace *stack = stack_buffer.data(); + stack->Reset(); + stack->Unwind(pc, frame, uc, common_flags()->fast_unwind_on_fatal); + + // The second stack frame contains the failure __hwasan_check function, as + // we have a stack frame for the registers saved in __hwasan_tag_mismatch that + // we wish to ignore. This (currently) only occurs on AArch64, as x64 + // implementations use SIGTRAP to implement the failure, and thus do not go + // through the stack saver. + if (registers_frame && stack->trace && stack->size > 0) { + stack->trace++; + stack->size--; + } + + bool fatal = flags()->halt_on_error || !ai.recover; + ReportTagMismatch(stack, ai.addr, ai.size, ai.is_store, fatal, + registers_frame); +} + +void HwasanTagMismatch(uptr addr, uptr access_info, uptr *registers_frame, + size_t outsize) { + __hwasan::AccessInfo ai; + ai.is_store = access_info & 0x10; + ai.is_load = !ai.is_store; + ai.recover = access_info & 0x20; + ai.addr = addr; + if ((access_info & 0xf) == 0xf) + ai.size = outsize; + else + ai.size = 1 << (access_info & 0xf); + + HandleTagMismatch(ai, (uptr)__builtin_return_address(0), + (uptr)__builtin_frame_address(0), nullptr, registers_frame); + __builtin_unreachable(); +} + +Thread *GetCurrentThread() { + uptr *ThreadLongPtr = GetCurrentThreadLongPtr(); + if (UNLIKELY(*ThreadLongPtr == 0)) + return nullptr; + auto *R = (StackAllocationsRingBuffer *)ThreadLongPtr; + return hwasanThreadList().GetThreadByBufferAddress((uptr)R->Next()); +} + } // namespace __hwasan using namespace __hwasan; @@ -216,7 +280,7 @@ static void InitLoadedGlobals() { static void InitInstrumentation() { if (hwasan_instrumentation_inited) return; - InitPrctl(); + InitializeOsSupport(); if (!InitShadow()) { Printf("FATAL: HWAddressSanitizer cannot mmap the shadow memory.\n"); @@ -225,7 +289,6 @@ static void InitInstrumentation() { } InitThreads(); - hwasanThreadList().CreateCurrentThread(); hwasan_instrumentation_inited = 1; } @@ -495,7 +558,7 @@ void __hwasan_print_memory_usage() { Printf("%s\n", s.data()); } -static const u8 kFallbackTag = 0xBB; +static const u8 kFallbackTag = 0xBB & kTagMask; u8 __hwasan_generate_tag() { Thread *t = GetCurrentThread(); @@ -516,4 +579,12 @@ void __sanitizer_print_stack_trace() { GET_FATAL_STACK_TRACE_PC_BP(StackTrace::GetCurrentPc(), GET_CURRENT_FRAME()); stack.Print(); } + +// Entry point for interoperability between __hwasan_tag_mismatch (ASM) and the +// rest of the mismatch handling code (C++). +void __hwasan_tag_mismatch4(uptr addr, uptr access_info, uptr *registers_frame, + size_t outsize) { + __hwasan::HwasanTagMismatch(addr, access_info, registers_frame, outsize); +} + } // extern "C" diff --git a/libsanitizer/hwasan/hwasan.h b/libsanitizer/hwasan/hwasan.h index 8515df559f3..7338b696ad3 100644 --- a/libsanitizer/hwasan/hwasan.h +++ b/libsanitizer/hwasan/hwasan.h @@ -36,7 +36,10 @@ typedef u8 tag_t; -#if defined(__x86_64__) +#if defined(HWASAN_ALIASING_MODE) +# if !defined(__x86_64__) +# error Aliasing mode is only supported on x86_64 +# endif // Tags are done in middle bits using userspace aliasing. constexpr unsigned kAddressTagShift = 39; constexpr unsigned kTagBits = 3; @@ -49,12 +52,16 @@ constexpr unsigned kTagBits = 3; // simpler/faster shadow calculation. constexpr unsigned kTaggableRegionCheckShift = __sanitizer::Max(kAddressTagShift + kTagBits + 1U, 44U); +#elif defined(__x86_64__) +// Tags are done in upper bits using Intel LAM. +constexpr unsigned kAddressTagShift = 57; +constexpr unsigned kTagBits = 6; #else // TBI (Top Byte Ignore) feature of AArch64: bits [63:56] are ignored in address // translation and can be used to store a tag. constexpr unsigned kAddressTagShift = 56; constexpr unsigned kTagBits = 8; -#endif // defined(__x86_64__) +#endif // defined(HWASAN_ALIASING_MODE) // Mask for extracting tag bits from the lower 8 bits. constexpr uptr kTagMask = (1UL << kTagBits) - 1; @@ -95,7 +102,7 @@ extern bool hwasan_init_is_running; extern int hwasan_report_count; bool InitShadow(); -void InitPrctl(); +void InitializeOsSupport(); void InitThreads(); void InitializeInterceptors(); @@ -129,6 +136,7 @@ void InstallAtExitHandler(); void HwasanTSDInit(); void HwasanTSDThreadInit(); +void HwasanAtExit(); void HwasanOnDeadlySignal(int signo, void *info, void *context); @@ -138,6 +146,26 @@ void AppendToErrorMessageBuffer(const char *buffer); void AndroidTestTlsSlot(); +// This is a compiler-generated struct that can be shared between hwasan +// implementations. +struct AccessInfo { + uptr addr; + uptr size; + bool is_store; + bool is_load; + bool recover; +}; + +// Given access info and frame information, unwind the stack and report the tag +// mismatch. +void HandleTagMismatch(AccessInfo ai, uptr pc, uptr frame, void *uc, + uptr *registers_frame = nullptr); + +// This dispatches to HandleTagMismatch but sets up the AccessInfo, program +// counter, and frame pointer. +void HwasanTagMismatch(uptr addr, uptr access_info, uptr *registers_frame, + size_t outsize); + } // namespace __hwasan #define HWASAN_MALLOC_HOOK(ptr, size) \ @@ -175,4 +203,12 @@ typedef struct __hw_jmp_buf_struct __hw_jmp_buf[1]; typedef struct __hw_jmp_buf_struct __hw_sigjmp_buf[1]; #endif // HWASAN_WITH_INTERCEPTORS && __aarch64__ +#define ENSURE_HWASAN_INITED() \ + do { \ + CHECK(!hwasan_init_is_running); \ + if (!hwasan_inited) { \ + __hwasan_init(); \ + } \ + } while (0) + #endif // HWASAN_H diff --git a/libsanitizer/hwasan/hwasan_allocation_functions.cpp b/libsanitizer/hwasan/hwasan_allocation_functions.cpp new file mode 100644 index 00000000000..6c2a6077866 --- /dev/null +++ b/libsanitizer/hwasan/hwasan_allocation_functions.cpp @@ -0,0 +1,172 @@ +//===-- hwasan_allocation_functions.cpp -----------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// This file is a part of HWAddressSanitizer. +// +// Definitions for __sanitizer allocation functions. +// +//===----------------------------------------------------------------------===// + +#include "hwasan.h" +#include "interception/interception.h" +#include "sanitizer_common/sanitizer_allocator_interface.h" +#include "sanitizer_common/sanitizer_tls_get_addr.h" + +using namespace __hwasan; + +static uptr allocated_for_dlsym; +static const uptr kDlsymAllocPoolSize = 1024; +static uptr alloc_memory_for_dlsym[kDlsymAllocPoolSize]; + +static bool IsInDlsymAllocPool(const void *ptr) { + uptr off = (uptr)ptr - (uptr)alloc_memory_for_dlsym; + return off < sizeof(alloc_memory_for_dlsym); +} + +static void *AllocateFromLocalPool(uptr size_in_bytes) { + uptr size_in_words = RoundUpTo(size_in_bytes, kWordSize) / kWordSize; + void *mem = (void *)&alloc_memory_for_dlsym[allocated_for_dlsym]; + allocated_for_dlsym += size_in_words; + CHECK_LT(allocated_for_dlsym, kDlsymAllocPoolSize); + return mem; +} + +int __sanitizer_posix_memalign(void **memptr, uptr alignment, uptr size) { + GET_MALLOC_STACK_TRACE; + CHECK_NE(memptr, 0); + int res = hwasan_posix_memalign(memptr, alignment, size, &stack); + return res; +} + +void *__sanitizer_memalign(uptr alignment, uptr size) { + GET_MALLOC_STACK_TRACE; + return hwasan_memalign(alignment, size, &stack); +} + +void *__sanitizer_aligned_alloc(uptr alignment, uptr size) { + GET_MALLOC_STACK_TRACE; + return hwasan_aligned_alloc(alignment, size, &stack); +} + +void *__sanitizer___libc_memalign(uptr alignment, uptr size) { + GET_MALLOC_STACK_TRACE; + void *ptr = hwasan_memalign(alignment, size, &stack); + if (ptr) + DTLS_on_libc_memalign(ptr, size); + return ptr; +} + +void *__sanitizer_valloc(uptr size) { + GET_MALLOC_STACK_TRACE; + return hwasan_valloc(size, &stack); +} + +void *__sanitizer_pvalloc(uptr size) { + GET_MALLOC_STACK_TRACE; + return hwasan_pvalloc(size, &stack); +} + +void __sanitizer_free(void *ptr) { + GET_MALLOC_STACK_TRACE; + if (!ptr || UNLIKELY(IsInDlsymAllocPool(ptr))) + return; + hwasan_free(ptr, &stack); +} + +void __sanitizer_cfree(void *ptr) { + GET_MALLOC_STACK_TRACE; + if (!ptr || UNLIKELY(IsInDlsymAllocPool(ptr))) + return; + hwasan_free(ptr, &stack); +} + +uptr __sanitizer_malloc_usable_size(const void *ptr) { + return __sanitizer_get_allocated_size(ptr); +} + +struct __sanitizer_struct_mallinfo __sanitizer_mallinfo() { + __sanitizer_struct_mallinfo sret; + internal_memset(&sret, 0, sizeof(sret)); + return sret; +} + +int __sanitizer_mallopt(int cmd, int value) { return 0; } + +void __sanitizer_malloc_stats(void) { + // FIXME: implement, but don't call REAL(malloc_stats)! +} + +void *__sanitizer_calloc(uptr nmemb, uptr size) { + GET_MALLOC_STACK_TRACE; + if (UNLIKELY(!hwasan_inited)) + // Hack: dlsym calls calloc before REAL(calloc) is retrieved from dlsym. + return AllocateFromLocalPool(nmemb * size); + return hwasan_calloc(nmemb, size, &stack); +} + +void *__sanitizer_realloc(void *ptr, uptr size) { + GET_MALLOC_STACK_TRACE; + if (UNLIKELY(IsInDlsymAllocPool(ptr))) { + uptr offset = (uptr)ptr - (uptr)alloc_memory_for_dlsym; + uptr copy_size = Min(size, kDlsymAllocPoolSize - offset); + void *new_ptr; + if (UNLIKELY(!hwasan_inited)) { + new_ptr = AllocateFromLocalPool(copy_size); + } else { + copy_size = size; + new_ptr = hwasan_malloc(copy_size, &stack); + } + internal_memcpy(new_ptr, ptr, copy_size); + return new_ptr; + } + return hwasan_realloc(ptr, size, &stack); +} + +void *__sanitizer_reallocarray(void *ptr, uptr nmemb, uptr size) { + GET_MALLOC_STACK_TRACE; + return hwasan_reallocarray(ptr, nmemb, size, &stack); +} + +void *__sanitizer_malloc(uptr size) { + GET_MALLOC_STACK_TRACE; + if (UNLIKELY(!hwasan_init_is_running)) + ENSURE_HWASAN_INITED(); + if (UNLIKELY(!hwasan_inited)) + // Hack: dlsym calls malloc before REAL(malloc) is retrieved from dlsym. + return AllocateFromLocalPool(size); + return hwasan_malloc(size, &stack); +} + +#if HWASAN_WITH_INTERCEPTORS +# define INTERCEPTOR_ALIAS(RET, FN, ARGS...) \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE RET WRAP(FN)(ARGS) \ + ALIAS("__sanitizer_" #FN); \ + extern "C" SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE RET FN( \ + ARGS) ALIAS("__sanitizer_" #FN) + +INTERCEPTOR_ALIAS(int, posix_memalign, void **memptr, SIZE_T alignment, + SIZE_T size); +INTERCEPTOR_ALIAS(void *, aligned_alloc, SIZE_T alignment, SIZE_T size); +INTERCEPTOR_ALIAS(void *, __libc_memalign, SIZE_T alignment, SIZE_T size); +INTERCEPTOR_ALIAS(void *, valloc, SIZE_T size); +INTERCEPTOR_ALIAS(void, free, void *ptr); +INTERCEPTOR_ALIAS(uptr, malloc_usable_size, const void *ptr); +INTERCEPTOR_ALIAS(void *, calloc, SIZE_T nmemb, SIZE_T size); +INTERCEPTOR_ALIAS(void *, realloc, void *ptr, SIZE_T size); +INTERCEPTOR_ALIAS(void *, reallocarray, void *ptr, SIZE_T nmemb, SIZE_T size); +INTERCEPTOR_ALIAS(void *, malloc, SIZE_T size); + +# if !SANITIZER_FREEBSD && !SANITIZER_NETBSD +INTERCEPTOR_ALIAS(void *, memalign, SIZE_T alignment, SIZE_T size); +INTERCEPTOR_ALIAS(void *, pvalloc, SIZE_T size); +INTERCEPTOR_ALIAS(void, cfree, void *ptr); +INTERCEPTOR_ALIAS(__sanitizer_struct_mallinfo, mallinfo); +INTERCEPTOR_ALIAS(int, mallopt, int cmd, int value); +INTERCEPTOR_ALIAS(void, malloc_stats, void); +# endif +#endif // #if HWASAN_WITH_INTERCEPTORS diff --git a/libsanitizer/hwasan/hwasan_allocator.cpp b/libsanitizer/hwasan/hwasan_allocator.cpp index a6fc794082a..ef6d4d6c767 100644 --- a/libsanitizer/hwasan/hwasan_allocator.cpp +++ b/libsanitizer/hwasan/hwasan_allocator.cpp @@ -80,12 +80,29 @@ void GetAllocatorStats(AllocatorStatCounters s) { allocator.GetStats(s); } +uptr GetAliasRegionStart() { +#if defined(HWASAN_ALIASING_MODE) + constexpr uptr kAliasRegionOffset = 1ULL << (kTaggableRegionCheckShift - 1); + uptr AliasRegionStart = + __hwasan_shadow_memory_dynamic_address + kAliasRegionOffset; + + CHECK_EQ(AliasRegionStart >> kTaggableRegionCheckShift, + __hwasan_shadow_memory_dynamic_address >> kTaggableRegionCheckShift); + CHECK_EQ( + (AliasRegionStart + kAliasRegionOffset - 1) >> kTaggableRegionCheckShift, + __hwasan_shadow_memory_dynamic_address >> kTaggableRegionCheckShift); + return AliasRegionStart; +#else + return 0; +#endif +} + void HwasanAllocatorInit() { atomic_store_relaxed(&hwasan_allocator_tagging_enabled, !flags()->disable_allocator_tagging); SetAllocatorMayReturnNull(common_flags()->allocator_may_return_null); allocator.Init(common_flags()->allocator_release_to_os_interval_ms, - kAliasRegionStart); + GetAliasRegionStart()); for (uptr i = 0; i < sizeof(tail_magic); i++) tail_magic[i] = GetCurrentThread()->GenerateRandomTag(); } @@ -196,6 +213,7 @@ static void HwasanDeallocate(StackTrace *stack, void *tagged_ptr) { : tagged_ptr; void *aligned_ptr = reinterpret_cast( RoundDownTo(reinterpret_cast(untagged_ptr), kShadowAlignment)); + tag_t pointer_tag = GetTagFromPointer(reinterpret_cast(tagged_ptr)); Metadata *meta = reinterpret_cast(allocator.GetMetaData(aligned_ptr)); uptr orig_size = meta->get_requested_size(); @@ -229,7 +247,20 @@ static void HwasanDeallocate(StackTrace *stack, void *tagged_ptr) { flags()->tag_in_free && malloc_bisect(stack, 0) && atomic_load_relaxed(&hwasan_allocator_tagging_enabled)) { // Always store full 8-bit tags on free to maximize UAF detection. - tag_t tag = t ? t->GenerateRandomTag(/*num_bits=*/8) : kFallbackFreeTag; + tag_t tag; + if (t) { + // Make sure we are not using a short granule tag as a poison tag. This + // would make us attempt to read the memory on a UaF. + // The tag can be zero if tagging is disabled on this thread. + do { + tag = t->GenerateRandomTag(/*num_bits=*/8); + } while ( + UNLIKELY((tag < kShadowAlignment || tag == pointer_tag) && tag != 0)); + } else { + static_assert(kFallbackFreeTag >= kShadowAlignment, + "fallback tag must not be a short granule tag."); + tag = kFallbackFreeTag; + } TagMemoryAligned(reinterpret_cast(aligned_ptr), TaggedSize(orig_size), tag); } diff --git a/libsanitizer/hwasan/hwasan_allocator.h b/libsanitizer/hwasan/hwasan_allocator.h index 03bbcff3f0f..35c3d6b4bf4 100644 --- a/libsanitizer/hwasan/hwasan_allocator.h +++ b/libsanitizer/hwasan/hwasan_allocator.h @@ -15,6 +15,7 @@ #include "hwasan.h" #include "hwasan_interface_internal.h" +#include "hwasan_mapping.h" #include "hwasan_poisoning.h" #include "sanitizer_common/sanitizer_allocator.h" #include "sanitizer_common/sanitizer_allocator_checks.h" @@ -58,7 +59,7 @@ static const uptr kMaxAllowedMallocSize = 1UL << 40; // 1T struct AP64 { static const uptr kSpaceBeg = ~0ULL; -#if defined(__x86_64__) +#if defined(HWASAN_ALIASING_MODE) static const uptr kSpaceSize = 1ULL << kAddressTagShift; #else static const uptr kSpaceSize = 0x2000000000ULL; @@ -110,11 +111,11 @@ typedef RingBuffer HeapAllocationsRingBuffer; void GetAllocatorStats(AllocatorStatCounters s); inline bool InTaggableRegion(uptr addr) { -#if defined(__x86_64__) +#if defined(HWASAN_ALIASING_MODE) // Aliases are mapped next to shadow so that the upper bits match the shadow // base. return (addr >> kTaggableRegionCheckShift) == - (__hwasan_shadow_memory_dynamic_address >> kTaggableRegionCheckShift); + (GetShadowOffset() >> kTaggableRegionCheckShift); #endif return true; } diff --git a/libsanitizer/hwasan/hwasan_dynamic_shadow.cpp b/libsanitizer/hwasan/hwasan_dynamic_shadow.cpp index f53276e330d..bde22dfa4bc 100644 --- a/libsanitizer/hwasan/hwasan_dynamic_shadow.cpp +++ b/libsanitizer/hwasan/hwasan_dynamic_shadow.cpp @@ -119,12 +119,12 @@ namespace __hwasan { void InitShadowGOT() {} uptr FindDynamicShadowStart(uptr shadow_size_bytes) { -#if defined(__x86_64__) +# if defined(HWASAN_ALIASING_MODE) constexpr uptr kAliasSize = 1ULL << kAddressTagShift; constexpr uptr kNumAliases = 1ULL << kTagBits; return MapDynamicShadowAndAliases(shadow_size_bytes, kAliasSize, kNumAliases, RingBufferSize()); -#endif +# endif return MapDynamicShadow(shadow_size_bytes, kShadowScale, kShadowBaseAlignment, kHighMemEnd); } diff --git a/libsanitizer/hwasan/hwasan_fuchsia.cpp b/libsanitizer/hwasan/hwasan_fuchsia.cpp new file mode 100644 index 00000000000..e61f6ada72f --- /dev/null +++ b/libsanitizer/hwasan/hwasan_fuchsia.cpp @@ -0,0 +1,192 @@ +//===-- hwasan_fuchsia.cpp --------------------------------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +/// +/// \file +/// This file is a part of HWAddressSanitizer and contains Fuchsia-specific +/// code. +/// +//===----------------------------------------------------------------------===// + +#include "sanitizer_common/sanitizer_fuchsia.h" +#if SANITIZER_FUCHSIA + +#include "hwasan.h" +#include "hwasan_interface_internal.h" +#include "hwasan_report.h" +#include "hwasan_thread.h" +#include "hwasan_thread_list.h" + +// This TLS variable contains the location of the stack ring buffer and can be +// used to always find the hwasan thread object associated with the current +// running thread. +[[gnu::tls_model("initial-exec")]] +SANITIZER_INTERFACE_ATTRIBUTE +THREADLOCAL uptr __hwasan_tls; + +namespace __hwasan { + +bool InitShadow() { + __sanitizer::InitShadowBounds(); + CHECK_NE(__sanitizer::ShadowBounds.shadow_limit, 0); + + return true; +} + +bool MemIsApp(uptr p) { + CHECK(GetTagFromPointer(p) == 0); + return __sanitizer::ShadowBounds.shadow_limit <= p && + p <= (__sanitizer::ShadowBounds.memory_limit - 1); +} + +// These are known parameters passed to the hwasan runtime on thread creation. +struct Thread::InitState { + uptr stack_bottom, stack_top; +}; + +static void FinishThreadInitialization(Thread *thread); + +void InitThreads() { + // This is the minimal alignment needed for the storage where hwasan threads + // and their stack ring buffers are placed. This alignment is necessary so the + // stack ring buffer can perform a simple calculation to get the next element + // in the RB. The instructions for this calculation are emitted by the + // compiler. (Full explanation in hwasan_thread_list.h.) + uptr alloc_size = UINT64_C(1) << kShadowBaseAlignment; + uptr thread_start = reinterpret_cast( + MmapAlignedOrDieOnFatalError(alloc_size, alloc_size, __func__)); + + InitThreadList(thread_start, alloc_size); + + // Create the hwasan thread object for the current (main) thread. Stack info + // for this thread is known from information passed via + // __sanitizer_startup_hook. + const Thread::InitState state = { + .stack_bottom = __sanitizer::MainThreadStackBase, + .stack_top = + __sanitizer::MainThreadStackBase + __sanitizer::MainThreadStackSize, + }; + FinishThreadInitialization(hwasanThreadList().CreateCurrentThread(&state)); +} + +uptr *GetCurrentThreadLongPtr() { return &__hwasan_tls; } + +// This is called from the parent thread before the new thread is created. Here +// we can propagate known info like the stack bounds to Thread::Init before +// jumping into the thread. We cannot initialize the stack ring buffer yet since +// we have not entered the new thread. +static void *BeforeThreadCreateHook(uptr user_id, bool detached, + const char *name, uptr stack_bottom, + uptr stack_size) { + const Thread::InitState state = { + .stack_bottom = stack_bottom, + .stack_top = stack_bottom + stack_size, + }; + return hwasanThreadList().CreateCurrentThread(&state); +} + +// This sets the stack top and bottom according to the InitState passed to +// CreateCurrentThread above. +void Thread::InitStackAndTls(const InitState *state) { + CHECK_NE(state->stack_bottom, 0); + CHECK_NE(state->stack_top, 0); + stack_bottom_ = state->stack_bottom; + stack_top_ = state->stack_top; + tls_end_ = tls_begin_ = 0; +} + +// This is called after creating a new thread with the pointer returned by +// BeforeThreadCreateHook. We are still in the creating thread and should check +// if it was actually created correctly. +static void ThreadCreateHook(void *hook, bool aborted) { + Thread *thread = static_cast(hook); + if (!aborted) { + // The thread was created successfully. + // ThreadStartHook can already be running in the new thread. + } else { + // The thread wasn't created after all. + // Clean up everything we set up in BeforeThreadCreateHook. + atomic_signal_fence(memory_order_seq_cst); + hwasanThreadList().ReleaseThread(thread); + } +} + +// This is called in the newly-created thread before it runs anything else, +// with the pointer returned by BeforeThreadCreateHook (above). Here we can +// setup the stack ring buffer. +static void ThreadStartHook(void *hook, thrd_t self) { + Thread *thread = static_cast(hook); + FinishThreadInitialization(thread); + thread->InitRandomState(); +} + +// This is the function that sets up the stack ring buffer and enables us to use +// GetCurrentThread. This function should only be called while IN the thread +// that we want to create the hwasan thread object for so __hwasan_tls can be +// properly referenced. +static void FinishThreadInitialization(Thread *thread) { + CHECK_NE(thread, nullptr); + + // The ring buffer is located immediately before the thread object. + uptr stack_buffer_size = hwasanThreadList().GetRingBufferSize(); + uptr stack_buffer_start = reinterpret_cast(thread) - stack_buffer_size; + thread->InitStackRingBuffer(stack_buffer_start, stack_buffer_size); +} + +static void ThreadExitHook(void *hook, thrd_t self) { + Thread *thread = static_cast(hook); + atomic_signal_fence(memory_order_seq_cst); + hwasanThreadList().ReleaseThread(thread); +} + +// Not implemented because Fuchsia does not use signal handlers. +void HwasanOnDeadlySignal(int signo, void *info, void *context) {} + +// Not implemented because Fuchsia does not use interceptors. +void InitializeInterceptors() {} + +// Not implemented because this is only relevant for Android. +void AndroidTestTlsSlot() {} + +// TSD was normally used on linux as a means of calling the hwasan thread exit +// handler passed to pthread_key_create. This is not needed on Fuchsia because +// we will be using __sanitizer_thread_exit_hook. +void HwasanTSDInit() {} +void HwasanTSDThreadInit() {} + +// On linux, this just would call `atexit(HwasanAtExit)`. The functions in +// HwasanAtExit are unimplemented for Fuchsia and effectively no-ops, so this +// function is unneeded. +void InstallAtExitHandler() {} + +} // namespace __hwasan + +extern "C" { + +void *__sanitizer_before_thread_create_hook(thrd_t thread, bool detached, + const char *name, void *stack_base, + size_t stack_size) { + return __hwasan::BeforeThreadCreateHook( + reinterpret_cast(thread), detached, name, + reinterpret_cast(stack_base), stack_size); +} + +void __sanitizer_thread_create_hook(void *hook, thrd_t thread, int error) { + __hwasan::ThreadCreateHook(hook, error != thrd_success); +} + +void __sanitizer_thread_start_hook(void *hook, thrd_t self) { + __hwasan::ThreadStartHook(hook, reinterpret_cast(self)); +} + +void __sanitizer_thread_exit_hook(void *hook, thrd_t self) { + __hwasan::ThreadExitHook(hook, self); +} + +} // extern "C" + +#endif // SANITIZER_FUCHSIA diff --git a/libsanitizer/hwasan/hwasan_interceptors.cpp b/libsanitizer/hwasan/hwasan_interceptors.cpp index ad67e2787d3..68f8adec077 100644 --- a/libsanitizer/hwasan/hwasan_interceptors.cpp +++ b/libsanitizer/hwasan/hwasan_interceptors.cpp @@ -16,192 +16,14 @@ #include "interception/interception.h" #include "hwasan.h" -#include "hwasan_allocator.h" -#include "hwasan_mapping.h" #include "hwasan_thread.h" -#include "hwasan_poisoning.h" -#include "hwasan_report.h" -#include "sanitizer_common/sanitizer_platform_limits_posix.h" -#include "sanitizer_common/sanitizer_allocator.h" -#include "sanitizer_common/sanitizer_allocator_interface.h" -#include "sanitizer_common/sanitizer_allocator_internal.h" -#include "sanitizer_common/sanitizer_atomic.h" -#include "sanitizer_common/sanitizer_common.h" -#include "sanitizer_common/sanitizer_errno.h" #include "sanitizer_common/sanitizer_stackdepot.h" -#include "sanitizer_common/sanitizer_libc.h" -#include "sanitizer_common/sanitizer_linux.h" -#include "sanitizer_common/sanitizer_tls_get_addr.h" -#include -// ACHTUNG! No other system header includes in this file. -// Ideally, we should get rid of stdarg.h as well. +#if !SANITIZER_FUCHSIA using namespace __hwasan; -using __sanitizer::memory_order; -using __sanitizer::atomic_load; -using __sanitizer::atomic_store; -using __sanitizer::atomic_uintptr_t; - -static uptr allocated_for_dlsym; -static const uptr kDlsymAllocPoolSize = 1024; -static uptr alloc_memory_for_dlsym[kDlsymAllocPoolSize]; - -static bool IsInDlsymAllocPool(const void *ptr) { - uptr off = (uptr)ptr - (uptr)alloc_memory_for_dlsym; - return off < sizeof(alloc_memory_for_dlsym); -} - -static void *AllocateFromLocalPool(uptr size_in_bytes) { - uptr size_in_words = RoundUpTo(size_in_bytes, kWordSize) / kWordSize; - void *mem = (void *)&alloc_memory_for_dlsym[allocated_for_dlsym]; - allocated_for_dlsym += size_in_words; - CHECK_LT(allocated_for_dlsym, kDlsymAllocPoolSize); - return mem; -} - -#define ENSURE_HWASAN_INITED() do { \ - CHECK(!hwasan_init_is_running); \ - if (!hwasan_inited) { \ - __hwasan_init(); \ - } \ -} while (0) - - -int __sanitizer_posix_memalign(void **memptr, uptr alignment, uptr size) { - GET_MALLOC_STACK_TRACE; - CHECK_NE(memptr, 0); - int res = hwasan_posix_memalign(memptr, alignment, size, &stack); - return res; -} - -void * __sanitizer_memalign(uptr alignment, uptr size) { - GET_MALLOC_STACK_TRACE; - return hwasan_memalign(alignment, size, &stack); -} - -void * __sanitizer_aligned_alloc(uptr alignment, uptr size) { - GET_MALLOC_STACK_TRACE; - return hwasan_aligned_alloc(alignment, size, &stack); -} - -void * __sanitizer___libc_memalign(uptr alignment, uptr size) { - GET_MALLOC_STACK_TRACE; - void *ptr = hwasan_memalign(alignment, size, &stack); - if (ptr) - DTLS_on_libc_memalign(ptr, size); - return ptr; -} - -void * __sanitizer_valloc(uptr size) { - GET_MALLOC_STACK_TRACE; - return hwasan_valloc(size, &stack); -} - -void * __sanitizer_pvalloc(uptr size) { - GET_MALLOC_STACK_TRACE; - return hwasan_pvalloc(size, &stack); -} - -void __sanitizer_free(void *ptr) { - GET_MALLOC_STACK_TRACE; - if (!ptr || UNLIKELY(IsInDlsymAllocPool(ptr))) return; - hwasan_free(ptr, &stack); -} - -void __sanitizer_cfree(void *ptr) { - GET_MALLOC_STACK_TRACE; - if (!ptr || UNLIKELY(IsInDlsymAllocPool(ptr))) return; - hwasan_free(ptr, &stack); -} - -uptr __sanitizer_malloc_usable_size(const void *ptr) { - return __sanitizer_get_allocated_size(ptr); -} - -struct __sanitizer_struct_mallinfo __sanitizer_mallinfo() { - __sanitizer_struct_mallinfo sret; - internal_memset(&sret, 0, sizeof(sret)); - return sret; -} - -int __sanitizer_mallopt(int cmd, int value) { - return 0; -} - -void __sanitizer_malloc_stats(void) { - // FIXME: implement, but don't call REAL(malloc_stats)! -} - -void * __sanitizer_calloc(uptr nmemb, uptr size) { - GET_MALLOC_STACK_TRACE; - if (UNLIKELY(!hwasan_inited)) - // Hack: dlsym calls calloc before REAL(calloc) is retrieved from dlsym. - return AllocateFromLocalPool(nmemb * size); - return hwasan_calloc(nmemb, size, &stack); -} - -void * __sanitizer_realloc(void *ptr, uptr size) { - GET_MALLOC_STACK_TRACE; - if (UNLIKELY(IsInDlsymAllocPool(ptr))) { - uptr offset = (uptr)ptr - (uptr)alloc_memory_for_dlsym; - uptr copy_size = Min(size, kDlsymAllocPoolSize - offset); - void *new_ptr; - if (UNLIKELY(!hwasan_inited)) { - new_ptr = AllocateFromLocalPool(copy_size); - } else { - copy_size = size; - new_ptr = hwasan_malloc(copy_size, &stack); - } - internal_memcpy(new_ptr, ptr, copy_size); - return new_ptr; - } - return hwasan_realloc(ptr, size, &stack); -} - -void * __sanitizer_reallocarray(void *ptr, uptr nmemb, uptr size) { - GET_MALLOC_STACK_TRACE; - return hwasan_reallocarray(ptr, nmemb, size, &stack); -} - -void * __sanitizer_malloc(uptr size) { - GET_MALLOC_STACK_TRACE; - if (UNLIKELY(!hwasan_init_is_running)) - ENSURE_HWASAN_INITED(); - if (UNLIKELY(!hwasan_inited)) - // Hack: dlsym calls malloc before REAL(malloc) is retrieved from dlsym. - return AllocateFromLocalPool(size); - return hwasan_malloc(size, &stack); -} - #if HWASAN_WITH_INTERCEPTORS -#define INTERCEPTOR_ALIAS(RET, FN, ARGS...) \ - extern "C" SANITIZER_INTERFACE_ATTRIBUTE RET WRAP(FN)(ARGS) \ - ALIAS("__sanitizer_" #FN); \ - extern "C" SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE RET FN( \ - ARGS) ALIAS("__sanitizer_" #FN) - -INTERCEPTOR_ALIAS(int, posix_memalign, void **memptr, SIZE_T alignment, - SIZE_T size); -INTERCEPTOR_ALIAS(void *, aligned_alloc, SIZE_T alignment, SIZE_T size); -INTERCEPTOR_ALIAS(void *, __libc_memalign, SIZE_T alignment, SIZE_T size); -INTERCEPTOR_ALIAS(void *, valloc, SIZE_T size); -INTERCEPTOR_ALIAS(void, free, void *ptr); -INTERCEPTOR_ALIAS(uptr, malloc_usable_size, const void *ptr); -INTERCEPTOR_ALIAS(void *, calloc, SIZE_T nmemb, SIZE_T size); -INTERCEPTOR_ALIAS(void *, realloc, void *ptr, SIZE_T size); -INTERCEPTOR_ALIAS(void *, reallocarray, void *ptr, SIZE_T nmemb, SIZE_T size); -INTERCEPTOR_ALIAS(void *, malloc, SIZE_T size); - -#if !SANITIZER_FREEBSD && !SANITIZER_NETBSD -INTERCEPTOR_ALIAS(void *, memalign, SIZE_T alignment, SIZE_T size); -INTERCEPTOR_ALIAS(void *, pvalloc, SIZE_T size); -INTERCEPTOR_ALIAS(void, cfree, void *ptr); -INTERCEPTOR_ALIAS(__sanitizer_struct_mallinfo, mallinfo); -INTERCEPTOR_ALIAS(int, mallopt, int cmd, int value); -INTERCEPTOR_ALIAS(void, malloc_stats, void); -#endif struct ThreadStartArg { thread_callback_t callback; @@ -346,3 +168,5 @@ void InitializeInterceptors() { inited = 1; } } // namespace __hwasan + +#endif // #if !SANITIZER_FUCHSIA diff --git a/libsanitizer/hwasan/hwasan_linux.cpp b/libsanitizer/hwasan/hwasan_linux.cpp index 8ce0ff7da95..e22723529f4 100644 --- a/libsanitizer/hwasan/hwasan_linux.cpp +++ b/libsanitizer/hwasan/hwasan_linux.cpp @@ -69,15 +69,9 @@ static void ProtectGap(uptr addr, uptr size) { uptr kLowMemStart; uptr kLowMemEnd; -uptr kLowShadowEnd; -uptr kLowShadowStart; -uptr kHighShadowStart; -uptr kHighShadowEnd; uptr kHighMemStart; uptr kHighMemEnd; -uptr kAliasRegionStart; // Always 0 on non-x86. - static void PrintRange(uptr start, uptr end, const char *name) { Printf("|| [%p, %p] || %.*s ||\n", (void *)start, (void *)end, 10, name); } @@ -116,7 +110,7 @@ static void InitializeShadowBaseAddress(uptr shadow_size_bytes) { FindDynamicShadowStart(shadow_size_bytes); } -void InitPrctl() { +void InitializeOsSupport() { #define PR_SET_TAGGED_ADDR_CTRL 55 #define PR_GET_TAGGED_ADDR_CTRL 56 #define PR_TAGGED_ADDR_ENABLE (1UL << 0) @@ -125,33 +119,50 @@ void InitPrctl() { if (internal_iserror(internal_prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0), &local_errno) && local_errno == EINVAL) { -#if SANITIZER_ANDROID || defined(__x86_64__) +# if SANITIZER_ANDROID || defined(HWASAN_ALIASING_MODE) // Some older Android kernels have the tagged pointer ABI on // unconditionally, and hence don't have the tagged-addr prctl while still // allow the ABI. // If targeting Android and the prctl is not around we assume this is the // case. return; -#else +# else if (flags()->fail_without_syscall_abi) { Printf( "FATAL: " "HWAddressSanitizer requires a kernel with tagged address ABI.\n"); Die(); } -#endif +# endif } // Turn on the tagged address ABI. if ((internal_iserror(internal_prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0)) || - !internal_prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0)) && - flags()->fail_without_syscall_abi) { - Printf( - "FATAL: HWAddressSanitizer failed to enable tagged address syscall " - "ABI.\nSuggest check `sysctl abi.tagged_addr_disabled` " - "configuration.\n"); - Die(); + !internal_prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0))) { +# if defined(__x86_64__) && !defined(HWASAN_ALIASING_MODE) + // Try the new prctl API for Intel LAM. The API is based on a currently + // unsubmitted patch to the Linux kernel (as of May 2021) and is thus + // subject to change. Patch is here: + // https://lore.kernel.org/linux-mm/20210205151631.43511-12-kirill.shutemov@linux.intel.com/ + int tag_bits = kTagBits; + int tag_shift = kAddressTagShift; + if (!internal_iserror( + internal_prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, + reinterpret_cast(&tag_bits), + reinterpret_cast(&tag_shift), 0))) { + CHECK_EQ(tag_bits, kTagBits); + CHECK_EQ(tag_shift, kAddressTagShift); + return; + } +# endif // defined(__x86_64__) && !defined(HWASAN_ALIASING_MODE) + if (flags()->fail_without_syscall_abi) { + Printf( + "FATAL: HWAddressSanitizer failed to enable tagged address syscall " + "ABI.\nSuggest check `sysctl abi.tagged_addr_disabled` " + "configuration.\n"); + Die(); + } } #undef PR_SET_TAGGED_ADDR_CTRL #undef PR_GET_TAGGED_ADDR_CTRL @@ -181,18 +192,6 @@ bool InitShadow() { // High memory starts where allocated shadow allows. kHighMemStart = ShadowToMem(kHighShadowStart); -#if defined(__x86_64__) - constexpr uptr kAliasRegionOffset = 1ULL << (kTaggableRegionCheckShift - 1); - kAliasRegionStart = - __hwasan_shadow_memory_dynamic_address + kAliasRegionOffset; - - CHECK_EQ(kAliasRegionStart >> kTaggableRegionCheckShift, - __hwasan_shadow_memory_dynamic_address >> kTaggableRegionCheckShift); - CHECK_EQ( - (kAliasRegionStart + kAliasRegionOffset - 1) >> kTaggableRegionCheckShift, - __hwasan_shadow_memory_dynamic_address >> kTaggableRegionCheckShift); -#endif - // Check the sanity of the defined memory ranges (there might be gaps). CHECK_EQ(kHighMemStart % GetMmapGranularity(), 0); CHECK_GT(kHighMemStart, kHighShadowEnd); @@ -233,25 +232,16 @@ void InitThreads() { ProtectGap(thread_space_end, __hwasan_shadow_memory_dynamic_address - thread_space_end); InitThreadList(thread_space_start, thread_space_end - thread_space_start); + hwasanThreadList().CreateCurrentThread(); } bool MemIsApp(uptr p) { -#if !defined(__x86_64__) // Memory outside the alias range has non-zero tags. +// Memory outside the alias range has non-zero tags. +# if !defined(HWASAN_ALIASING_MODE) CHECK(GetTagFromPointer(p) == 0); -#endif - return p >= kHighMemStart || (p >= kLowMemStart && p <= kLowMemEnd); -} +# endif -static void HwasanAtExit(void) { - if (common_flags()->print_module_map) - DumpProcessMap(); - if (flags()->print_stats && (flags()->atexit || hwasan_report_count > 0)) - ReportStats(); - if (hwasan_report_count > 0) { - // ReportAtExitStatistics(); - if (common_flags()->exitcode) - internal__exit(common_flags()->exitcode); - } + return p >= kHighMemStart || (p >= kLowMemStart && p <= kLowMemEnd); } void InstallAtExitHandler() { @@ -330,22 +320,6 @@ void AndroidTestTlsSlot() { void AndroidTestTlsSlot() {} #endif -Thread *GetCurrentThread() { - uptr *ThreadLongPtr = GetCurrentThreadLongPtr(); - if (UNLIKELY(*ThreadLongPtr == 0)) - return nullptr; - auto *R = (StackAllocationsRingBuffer *)ThreadLongPtr; - return hwasanThreadList().GetThreadByBufferAddress((uptr)R->Next()); -} - -struct AccessInfo { - uptr addr; - uptr size; - bool is_store; - bool is_load; - bool recover; -}; - static AccessInfo GetAccessInfo(siginfo_t *info, ucontext_t *uc) { // Access type is passed in a platform dependent way (see below) and encoded // as 0xXY, where X&1 is 1 for store, 0 for load, and X&2 is 1 if the error is @@ -396,28 +370,6 @@ static AccessInfo GetAccessInfo(siginfo_t *info, ucontext_t *uc) { return AccessInfo{addr, size, is_store, !is_store, recover}; } -static void HandleTagMismatch(AccessInfo ai, uptr pc, uptr frame, - ucontext_t *uc, uptr *registers_frame = nullptr) { - InternalMmapVector stack_buffer(1); - BufferedStackTrace *stack = stack_buffer.data(); - stack->Reset(); - stack->Unwind(pc, frame, uc, common_flags()->fast_unwind_on_fatal); - - // The second stack frame contains the failure __hwasan_check function, as - // we have a stack frame for the registers saved in __hwasan_tag_mismatch that - // we wish to ignore. This (currently) only occurs on AArch64, as x64 - // implementations use SIGTRAP to implement the failure, and thus do not go - // through the stack saver. - if (registers_frame && stack->trace && stack->size > 0) { - stack->trace++; - stack->size--; - } - - bool fatal = flags()->halt_on_error || !ai.recover; - ReportTagMismatch(stack, ai.addr, ai.size, ai.is_store, fatal, - registers_frame); -} - static bool HwasanOnSIGTRAP(int signo, siginfo_t *info, ucontext_t *uc) { AccessInfo ai = GetAccessInfo(info, uc); if (!ai.is_store && !ai.is_load) @@ -450,27 +402,39 @@ void HwasanOnDeadlySignal(int signo, void *info, void *context) { HandleDeadlySignal(info, context, GetTid(), &OnStackUnwind, nullptr); } +void Thread::InitStackAndTls(const InitState *) { + uptr tls_size; + uptr stack_size; + GetThreadStackAndTls(IsMainThread(), &stack_bottom_, &stack_size, &tls_begin_, + &tls_size); + stack_top_ = stack_bottom_ + stack_size; + tls_end_ = tls_begin_ + tls_size; +} -} // namespace __hwasan - -// Entry point for interoperability between __hwasan_tag_mismatch (ASM) and the -// rest of the mismatch handling code (C++). -void __hwasan_tag_mismatch4(uptr addr, uptr access_info, uptr *registers_frame, - size_t outsize) { - __hwasan::AccessInfo ai; - ai.is_store = access_info & 0x10; - ai.is_load = !ai.is_store; - ai.recover = access_info & 0x20; - ai.addr = addr; - if ((access_info & 0xf) == 0xf) - ai.size = outsize; - else - ai.size = 1 << (access_info & 0xf); - - __hwasan::HandleTagMismatch(ai, (uptr)__builtin_return_address(0), - (uptr)__builtin_frame_address(0), nullptr, - registers_frame); - __builtin_unreachable(); +uptr TagMemoryAligned(uptr p, uptr size, tag_t tag) { + CHECK(IsAligned(p, kShadowAlignment)); + CHECK(IsAligned(size, kShadowAlignment)); + uptr shadow_start = MemToShadow(p); + uptr shadow_size = MemToShadowSize(size); + + uptr page_size = GetPageSizeCached(); + uptr page_start = RoundUpTo(shadow_start, page_size); + uptr page_end = RoundDownTo(shadow_start + shadow_size, page_size); + uptr threshold = common_flags()->clear_shadow_mmap_threshold; + if (SANITIZER_LINUX && + UNLIKELY(page_end >= page_start + threshold && tag == 0)) { + internal_memset((void *)shadow_start, tag, page_start - shadow_start); + internal_memset((void *)page_end, tag, + shadow_start + shadow_size - page_end); + // For an anonymous private mapping MADV_DONTNEED will return a zero page on + // Linux. + ReleaseMemoryPagesToOSAndZeroFill(page_start, page_end); + } else { + internal_memset((void *)shadow_start, tag, shadow_size); + } + return AddTagToPointer(p, tag); } +} // namespace __hwasan + #endif // SANITIZER_FREEBSD || SANITIZER_LINUX || SANITIZER_NETBSD diff --git a/libsanitizer/hwasan/hwasan_mapping.h b/libsanitizer/hwasan/hwasan_mapping.h index 8243d1ec7ed..79a143632f6 100644 --- a/libsanitizer/hwasan/hwasan_mapping.h +++ b/libsanitizer/hwasan/hwasan_mapping.h @@ -48,14 +48,14 @@ extern uptr kHighShadowEnd; extern uptr kHighMemStart; extern uptr kHighMemEnd; -extern uptr kAliasRegionStart; - +inline uptr GetShadowOffset() { + return SANITIZER_FUCHSIA ? 0 : __hwasan_shadow_memory_dynamic_address; +} inline uptr MemToShadow(uptr untagged_addr) { - return (untagged_addr >> kShadowScale) + - __hwasan_shadow_memory_dynamic_address; + return (untagged_addr >> kShadowScale) + GetShadowOffset(); } inline uptr ShadowToMem(uptr shadow_addr) { - return (shadow_addr - __hwasan_shadow_memory_dynamic_address) << kShadowScale; + return (shadow_addr - GetShadowOffset()) << kShadowScale; } inline uptr MemToShadowSize(uptr size) { return size >> kShadowScale; @@ -63,6 +63,13 @@ inline uptr MemToShadowSize(uptr size) { bool MemIsApp(uptr p); +inline bool MemIsShadow(uptr p) { + return (kLowShadowStart <= p && p <= kLowShadowEnd) || + (kHighShadowStart <= p && p <= kHighShadowEnd); +} + +uptr GetAliasRegionStart(); + } // namespace __hwasan #endif // HWASAN_MAPPING_H diff --git a/libsanitizer/hwasan/hwasan_new_delete.cpp b/libsanitizer/hwasan/hwasan_new_delete.cpp index 69cddda736e..4e057a651e1 100644 --- a/libsanitizer/hwasan/hwasan_new_delete.cpp +++ b/libsanitizer/hwasan/hwasan_new_delete.cpp @@ -56,7 +56,6 @@ using namespace __hwasan; // Fake std::nothrow_t to avoid including . namespace std { struct nothrow_t {}; - enum class align_val_t : size_t {}; } // namespace std @@ -73,6 +72,32 @@ INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void *operator new[](size_t size, std::nothrow_t const&) { OPERATOR_NEW_BODY(true /*nothrow*/); } + +INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete(void *ptr) + NOEXCEPT { + OPERATOR_DELETE_BODY; +} +INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete[]( + void *ptr) NOEXCEPT { + OPERATOR_DELETE_BODY; +} +INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete( + void *ptr, std::nothrow_t const &) { + OPERATOR_DELETE_BODY; +} +INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete[]( + void *ptr, std::nothrow_t const &) { + OPERATOR_DELETE_BODY; +} + +#endif // OPERATOR_NEW_BODY + +#ifdef OPERATOR_NEW_ALIGN_BODY + +namespace std { +enum class align_val_t : size_t {}; +} // namespace std + INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void *operator new( size_t size, std::align_val_t align) { OPERATOR_NEW_ALIGN_BODY(false /*nothrow*/); @@ -90,16 +115,6 @@ INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void *operator new[]( OPERATOR_NEW_ALIGN_BODY(true /*nothrow*/); } -INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void operator delete(void *ptr) NOEXCEPT { OPERATOR_DELETE_BODY; } -INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void operator delete[](void *ptr) NOEXCEPT { OPERATOR_DELETE_BODY; } -INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void operator delete(void *ptr, std::nothrow_t const&) { OPERATOR_DELETE_BODY; } -INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE -void operator delete[](void *ptr, std::nothrow_t const&) { - OPERATOR_DELETE_BODY; -} INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete( void *ptr, std::align_val_t align) NOEXCEPT { OPERATOR_DELETE_BODY; @@ -117,4 +132,4 @@ INTERCEPTOR_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void operator delete[]( OPERATOR_DELETE_BODY; } -#endif // OPERATOR_NEW_BODY +#endif // OPERATOR_NEW_ALIGN_BODY diff --git a/libsanitizer/hwasan/hwasan_poisoning.cpp b/libsanitizer/hwasan/hwasan_poisoning.cpp index 2a0816428e7..5aafdb1884b 100644 --- a/libsanitizer/hwasan/hwasan_poisoning.cpp +++ b/libsanitizer/hwasan/hwasan_poisoning.cpp @@ -19,30 +19,6 @@ namespace __hwasan { -uptr TagMemoryAligned(uptr p, uptr size, tag_t tag) { - CHECK(IsAligned(p, kShadowAlignment)); - CHECK(IsAligned(size, kShadowAlignment)); - uptr shadow_start = MemToShadow(p); - uptr shadow_size = MemToShadowSize(size); - - uptr page_size = GetPageSizeCached(); - uptr page_start = RoundUpTo(shadow_start, page_size); - uptr page_end = RoundDownTo(shadow_start + shadow_size, page_size); - uptr threshold = common_flags()->clear_shadow_mmap_threshold; - if (SANITIZER_LINUX && - UNLIKELY(page_end >= page_start + threshold && tag == 0)) { - internal_memset((void *)shadow_start, tag, page_start - shadow_start); - internal_memset((void *)page_end, tag, - shadow_start + shadow_size - page_end); - // For an anonymous private mapping MADV_DONTNEED will return a zero page on - // Linux. - ReleaseMemoryPagesToOSAndZeroFill(page_start, page_end); - } else { - internal_memset((void *)shadow_start, tag, shadow_size); - } - return AddTagToPointer(p, tag); -} - uptr TagMemory(uptr p, uptr size, tag_t tag) { uptr start = RoundDownTo(p, kShadowAlignment); uptr end = RoundUpTo(p + size, kShadowAlignment); diff --git a/libsanitizer/hwasan/hwasan_report.cpp b/libsanitizer/hwasan/hwasan_report.cpp index c0217799391..44047c9fdaf 100644 --- a/libsanitizer/hwasan/hwasan_report.cpp +++ b/libsanitizer/hwasan/hwasan_report.cpp @@ -236,12 +236,12 @@ static void PrintStackAllocations(StackAllocationsRingBuffer *sa, frame_desc.append(" record_addr:0x%zx record:0x%zx", reinterpret_cast(record_addr), record); if (SymbolizedStack *frame = Symbolizer::GetOrInit()->SymbolizePC(pc)) { - RenderFrame(&frame_desc, " %F %L\n", 0, frame->info.address, &frame->info, + RenderFrame(&frame_desc, " %F %L", 0, frame->info.address, &frame->info, common_flags()->symbolize_vs_style, common_flags()->strip_path_prefix); frame->ClearAll(); } - Printf("%s", frame_desc.data()); + Printf("%s\n", frame_desc.data()); frame_desc.clear(); } } @@ -296,6 +296,75 @@ static uptr GetGlobalSizeFromDescriptor(uptr ptr) { return 0; } +static void ShowHeapOrGlobalCandidate(uptr untagged_addr, tag_t *candidate, + tag_t *left, tag_t *right) { + Decorator d; + uptr mem = ShadowToMem(reinterpret_cast(candidate)); + HwasanChunkView chunk = FindHeapChunkByAddress(mem); + if (chunk.IsAllocated()) { + uptr offset; + const char *whence; + if (untagged_addr < chunk.End() && untagged_addr >= chunk.Beg()) { + offset = untagged_addr - chunk.Beg(); + whence = "inside"; + } else if (candidate == left) { + offset = untagged_addr - chunk.End(); + whence = "to the right of"; + } else { + offset = chunk.Beg() - untagged_addr; + whence = "to the left of"; + } + Printf("%s", d.Error()); + Printf("\nCause: heap-buffer-overflow\n"); + Printf("%s", d.Default()); + Printf("%s", d.Location()); + Printf("%p is located %zd bytes %s %zd-byte region [%p,%p)\n", + untagged_addr, offset, whence, chunk.UsedSize(), chunk.Beg(), + chunk.End()); + Printf("%s", d.Allocation()); + Printf("allocated here:\n"); + Printf("%s", d.Default()); + GetStackTraceFromId(chunk.GetAllocStackId()).Print(); + return; + } + // Check whether the address points into a loaded library. If so, this is + // most likely a global variable. + const char *module_name; + uptr module_address; + Symbolizer *sym = Symbolizer::GetOrInit(); + if (sym->GetModuleNameAndOffsetForPC(mem, &module_name, &module_address)) { + Printf("%s", d.Error()); + Printf("\nCause: global-overflow\n"); + Printf("%s", d.Default()); + DataInfo info; + Printf("%s", d.Location()); + if (sym->SymbolizeData(mem, &info) && info.start) { + Printf( + "%p is located %zd bytes to the %s of %zd-byte global variable " + "%s [%p,%p) in %s\n", + untagged_addr, + candidate == left ? untagged_addr - (info.start + info.size) + : info.start - untagged_addr, + candidate == left ? "right" : "left", info.size, info.name, + info.start, info.start + info.size, module_name); + } else { + uptr size = GetGlobalSizeFromDescriptor(mem); + if (size == 0) + // We couldn't find the size of the global from the descriptors. + Printf("%p is located to the %s of a global variable in (%s+0x%x)\n", + untagged_addr, candidate == left ? "right" : "left", module_name, + module_address); + else + Printf( + "%p is located to the %s of a %zd-byte global variable in " + "(%s+0x%x)\n", + untagged_addr, candidate == left ? "right" : "left", size, + module_name, module_address); + } + Printf("%s", d.Default()); + } +} + void PrintAddressDescription( uptr tagged_addr, uptr access_size, StackAllocationsRingBuffer *current_stack_allocations) { @@ -317,78 +386,59 @@ void PrintAddressDescription( d.Default()); } + tag_t addr_tag = GetTagFromPointer(tagged_addr); + + bool on_stack = false; + // Check stack first. If the address is on the stack of a live thread, we + // know it cannot be a heap / global overflow. + hwasanThreadList().VisitAllLiveThreads([&](Thread *t) { + if (t->AddrIsInStack(untagged_addr)) { + on_stack = true; + // TODO(fmayer): figure out how to distinguish use-after-return and + // stack-buffer-overflow. + Printf("%s", d.Error()); + Printf("\nCause: stack tag-mismatch\n"); + Printf("%s", d.Location()); + Printf("Address %p is located in stack of thread T%zd\n", untagged_addr, + t->unique_id()); + Printf("%s", d.Default()); + t->Announce(); + + auto *sa = (t == GetCurrentThread() && current_stack_allocations) + ? current_stack_allocations + : t->stack_allocations(); + PrintStackAllocations(sa, addr_tag, untagged_addr); + num_descriptions_printed++; + } + }); + // Check if this looks like a heap buffer overflow by scanning // the shadow left and right and looking for the first adjacent // object with a different memory tag. If that tag matches addr_tag, // check the allocator if it has a live chunk there. - tag_t addr_tag = GetTagFromPointer(tagged_addr); tag_t *tag_ptr = reinterpret_cast(MemToShadow(untagged_addr)); tag_t *candidate = nullptr, *left = tag_ptr, *right = tag_ptr; - for (int i = 0; i < 1000; i++) { - if (TagsEqual(addr_tag, left)) { + uptr candidate_distance = 0; + for (; candidate_distance < 1000; candidate_distance++) { + if (MemIsShadow(reinterpret_cast(left)) && + TagsEqual(addr_tag, left)) { candidate = left; break; } --left; - if (TagsEqual(addr_tag, right)) { + if (MemIsShadow(reinterpret_cast(right)) && + TagsEqual(addr_tag, right)) { candidate = right; break; } ++right; } - if (candidate) { - uptr mem = ShadowToMem(reinterpret_cast(candidate)); - HwasanChunkView chunk = FindHeapChunkByAddress(mem); - if (chunk.IsAllocated()) { - Printf("%s", d.Location()); - Printf("%p is located %zd bytes to the %s of %zd-byte region [%p,%p)\n", - untagged_addr, - candidate == left ? untagged_addr - chunk.End() - : chunk.Beg() - untagged_addr, - candidate == left ? "right" : "left", chunk.UsedSize(), - chunk.Beg(), chunk.End()); - Printf("%s", d.Allocation()); - Printf("allocated here:\n"); - Printf("%s", d.Default()); - GetStackTraceFromId(chunk.GetAllocStackId()).Print(); - num_descriptions_printed++; - } else { - // Check whether the address points into a loaded library. If so, this is - // most likely a global variable. - const char *module_name; - uptr module_address; - Symbolizer *sym = Symbolizer::GetOrInit(); - if (sym->GetModuleNameAndOffsetForPC(mem, &module_name, - &module_address)) { - DataInfo info; - if (sym->SymbolizeData(mem, &info) && info.start) { - Printf( - "%p is located %zd bytes to the %s of %zd-byte global variable " - "%s [%p,%p) in %s\n", - untagged_addr, - candidate == left ? untagged_addr - (info.start + info.size) - : info.start - untagged_addr, - candidate == left ? "right" : "left", info.size, info.name, - info.start, info.start + info.size, module_name); - } else { - uptr size = GetGlobalSizeFromDescriptor(mem); - if (size == 0) - // We couldn't find the size of the global from the descriptors. - Printf( - "%p is located to the %s of a global variable in (%s+0x%x)\n", - untagged_addr, candidate == left ? "right" : "left", - module_name, module_address); - else - Printf( - "%p is located to the %s of a %zd-byte global variable in " - "(%s+0x%x)\n", - untagged_addr, candidate == left ? "right" : "left", size, - module_name, module_address); - } - num_descriptions_printed++; - } - } + constexpr auto kCloseCandidateDistance = 1; + + if (!on_stack && candidate && candidate_distance <= kCloseCandidateDistance) { + ShowHeapOrGlobalCandidate(untagged_addr, candidate, left, right); + num_descriptions_printed++; } hwasanThreadList().VisitAllLiveThreads([&](Thread *t) { @@ -398,6 +448,8 @@ void PrintAddressDescription( if (FindHeapAllocation(t->heap_allocations(), tagged_addr, &har, &ring_index, &num_matching_addrs, &num_matching_addrs_4b)) { + Printf("%s", d.Error()); + Printf("\nCause: use-after-free\n"); Printf("%s", d.Location()); Printf("%p is located %zd bytes inside of %zd-byte region [%p,%p)\n", untagged_addr, untagged_addr - UntagAddr(har.tagged_addr), @@ -424,29 +476,25 @@ void PrintAddressDescription( t->Announce(); num_descriptions_printed++; } - - // Very basic check for stack memory. - if (t->AddrIsInStack(untagged_addr)) { - Printf("%s", d.Location()); - Printf("Address %p is located in stack of thread T%zd\n", untagged_addr, - t->unique_id()); - Printf("%s", d.Default()); - t->Announce(); - - auto *sa = (t == GetCurrentThread() && current_stack_allocations) - ? current_stack_allocations - : t->stack_allocations(); - PrintStackAllocations(sa, addr_tag, untagged_addr); - num_descriptions_printed++; - } }); + if (candidate && num_descriptions_printed == 0) { + ShowHeapOrGlobalCandidate(untagged_addr, candidate, left, right); + num_descriptions_printed++; + } + // Print the remaining threads, as an extra information, 1 line per thread. hwasanThreadList().VisitAllLiveThreads([&](Thread *t) { t->Announce(); }); if (!num_descriptions_printed) // We exhausted our possibilities. Bail out. Printf("HWAddressSanitizer can not describe address in more detail.\n"); + if (num_descriptions_printed > 1) { + Printf( + "There are %d potential causes, printed above in order " + "of likeliness.\n", + num_descriptions_printed); + } } void ReportStats() {} @@ -538,6 +586,12 @@ void ReportTailOverwritten(StackTrace *stack, uptr tagged_addr, uptr orig_size, Report("ERROR: %s: %s; heap object [%p,%p) of size %zd\n", SanitizerToolName, bug_type, untagged_addr, untagged_addr + orig_size, orig_size); Printf("\n%s", d.Default()); + Printf( + "Stack of invalid access unknown. Issue detected at deallocation " + "time.\n"); + Printf("%s", d.Allocation()); + Printf("deallocated here:\n"); + Printf("%s", d.Default()); stack->Print(); HwasanChunkView chunk = FindHeapChunkByAddress(untagged_addr); if (chunk.Beg()) { @@ -657,8 +711,10 @@ void ReportRegisters(uptr *frame, uptr pc) { frame[20], frame[21], frame[22], frame[23]); Printf(" x24 %016llx x25 %016llx x26 %016llx x27 %016llx\n", frame[24], frame[25], frame[26], frame[27]); - Printf(" x28 %016llx x29 %016llx x30 %016llx\n", - frame[28], frame[29], frame[30]); + // hwasan_check* reduces the stack pointer by 256, then __hwasan_tag_mismatch + // passes it to this function. + Printf(" x28 %016llx x29 %016llx x30 %016llx sp %016llx\n", frame[28], + frame[29], frame[30], reinterpret_cast(frame) + 256); } } // namespace __hwasan diff --git a/libsanitizer/hwasan/hwasan_thread.cpp b/libsanitizer/hwasan/hwasan_thread.cpp index bb4d56abed0..ee747a3beea 100644 --- a/libsanitizer/hwasan/hwasan_thread.cpp +++ b/libsanitizer/hwasan/hwasan_thread.cpp @@ -34,7 +34,8 @@ void Thread::InitRandomState() { stack_allocations_->push(0); } -void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size) { +void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size, + const InitState *state) { CHECK_EQ(0, unique_id_); // try to catch bad stack reuse CHECK_EQ(0, stack_top_); CHECK_EQ(0, stack_bottom_); @@ -44,6 +45,17 @@ void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size) { if (auto sz = flags()->heap_history_size) heap_allocations_ = HeapAllocationsRingBuffer::New(sz); + InitStackAndTls(state); +#if !SANITIZER_FUCHSIA + // Do not initialize the stack ring buffer just yet on Fuchsia. Threads will + // be initialized before we enter the thread itself, so we will instead call + // this later. + InitStackRingBuffer(stack_buffer_start, stack_buffer_size); +#endif +} + +void Thread::InitStackRingBuffer(uptr stack_buffer_start, + uptr stack_buffer_size) { HwasanTSDThreadInit(); // Only needed with interceptors. uptr *ThreadLong = GetCurrentThreadLongPtr(); // The following implicitly sets (this) as the current thread. @@ -55,13 +67,6 @@ void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size) { // ScopedTaggingDisable needs GetCurrentThread to be set up. ScopedTaggingDisabler disabler; - uptr tls_size; - uptr stack_size; - GetThreadStackAndTls(IsMainThread(), &stack_bottom_, &stack_size, &tls_begin_, - &tls_size); - stack_top_ = stack_bottom_ + stack_size; - tls_end_ = tls_begin_ + tls_size; - if (stack_bottom_) { int local; CHECK(AddrIsInStack((uptr)&local)); diff --git a/libsanitizer/hwasan/hwasan_thread.h b/libsanitizer/hwasan/hwasan_thread.h index 1c71cab41c4..9f20afe1dc7 100644 --- a/libsanitizer/hwasan/hwasan_thread.h +++ b/libsanitizer/hwasan/hwasan_thread.h @@ -23,8 +23,17 @@ typedef __sanitizer::CompactRingBuffer StackAllocationsRingBuffer; class Thread { public: - void Init(uptr stack_buffer_start, uptr stack_buffer_size); // Must be called from the thread itself. + // These are optional parameters that can be passed to Init. + struct InitState; + + void Init(uptr stack_buffer_start, uptr stack_buffer_size, + const InitState *state = nullptr); void InitRandomState(); + void InitStackAndTls(const InitState *state = nullptr); + + // Must be called from the thread itself. + void InitStackRingBuffer(uptr stack_buffer_start, uptr stack_buffer_size); + void Destroy(); uptr stack_top() { return stack_top_; } diff --git a/libsanitizer/hwasan/hwasan_thread_list.cpp b/libsanitizer/hwasan/hwasan_thread_list.cpp index a31eee84ed9..fa46e658b69 100644 --- a/libsanitizer/hwasan/hwasan_thread_list.cpp +++ b/libsanitizer/hwasan/hwasan_thread_list.cpp @@ -12,4 +12,4 @@ void InitThreadList(uptr storage, uptr size) { new (thread_list_placeholder) HwasanThreadList(storage, size); } -} // namespace +} // namespace __hwasan diff --git a/libsanitizer/hwasan/hwasan_thread_list.h b/libsanitizer/hwasan/hwasan_thread_list.h index 11c586314ce..15916a802d6 100644 --- a/libsanitizer/hwasan/hwasan_thread_list.h +++ b/libsanitizer/hwasan/hwasan_thread_list.h @@ -85,7 +85,7 @@ class HwasanThreadList { RoundUpTo(ring_buffer_size_ + sizeof(Thread), ring_buffer_size_ * 2); } - Thread *CreateCurrentThread() { + Thread *CreateCurrentThread(const Thread::InitState *state = nullptr) { Thread *t = nullptr; { SpinMutexLock l(&free_list_mutex_); @@ -104,7 +104,7 @@ class HwasanThreadList { SpinMutexLock l(&live_list_mutex_); live_list_.push_back(t); } - t->Init((uptr)t - ring_buffer_size_, ring_buffer_size_); + t->Init((uptr)t - ring_buffer_size_, ring_buffer_size_, state); AddThreadStats(t); return t; } @@ -171,6 +171,8 @@ class HwasanThreadList { return stats_; } + uptr GetRingBufferSize() const { return ring_buffer_size_; } + private: Thread *AllocThread() { SpinMutexLock l(&free_space_mutex_); @@ -200,4 +202,4 @@ class HwasanThreadList { void InitThreadList(uptr storage, uptr size); HwasanThreadList &hwasanThreadList(); -} // namespace +} // namespace __hwasan diff --git a/libsanitizer/include/sanitizer/dfsan_interface.h b/libsanitizer/include/sanitizer/dfsan_interface.h index 40f9379b557..cd3b6d6e2b1 100644 --- a/libsanitizer/include/sanitizer/dfsan_interface.h +++ b/libsanitizer/include/sanitizer/dfsan_interface.h @@ -21,34 +21,15 @@ extern "C" { #endif -typedef uint16_t dfsan_label; +typedef uint8_t dfsan_label; typedef uint32_t dfsan_origin; -/// Stores information associated with a specific label identifier. A label -/// may be a base label created using dfsan_create_label, with associated -/// text description and user data, or an automatically created union label, -/// which represents the union of two label identifiers (which may themselves -/// be base or union labels). -struct dfsan_label_info { - // Fields for union labels, set to 0 for base labels. - dfsan_label l1; - dfsan_label l2; - - // Fields for base labels. - const char *desc; - void *userdata; -}; - /// Signature of the callback argument to dfsan_set_write_callback(). typedef void (*dfsan_write_callback_t)(int fd, const void *buf, size_t count); -/// Computes the union of \c l1 and \c l2, possibly creating a union label in -/// the process. +/// Computes the union of \c l1 and \c l2, resulting in a union label. dfsan_label dfsan_union(dfsan_label l1, dfsan_label l2); -/// Creates and returns a base label with the given description and user data. -dfsan_label dfsan_create_label(const char *desc, void *userdata); - /// Sets the label for each address in [addr,addr+size) to \c label. void dfsan_set_label(dfsan_label label, void *addr, size_t size); @@ -73,19 +54,9 @@ dfsan_origin dfsan_get_origin(long data); /// Retrieves the label associated with the data at the given address. dfsan_label dfsan_read_label(const void *addr, size_t size); -/// Retrieves a pointer to the dfsan_label_info struct for the given label. -const struct dfsan_label_info *dfsan_get_label_info(dfsan_label label); - /// Returns whether the given label label contains the label elem. int dfsan_has_label(dfsan_label label, dfsan_label elem); -/// If the given label label contains a label with the description desc, returns -/// that label, else returns 0. -dfsan_label dfsan_has_label_with_desc(dfsan_label label, const char *desc); - -/// Returns the number of labels allocated. -size_t dfsan_get_label_count(void); - /// Flushes the DFSan shadow, i.e. forgets about all labels currently associated /// with the application memory. Use this call to start over the taint tracking /// within the same process. @@ -99,12 +70,6 @@ void dfsan_flush(void); /// callback executes. Pass in NULL to remove any callback. void dfsan_set_write_callback(dfsan_write_callback_t labeled_write_callback); -/// Writes the labels currently used by the program to the given file -/// descriptor. The lines of the output have the following format: -/// -///