feat: minimal tombstone integration #4933

supervacuus · 2025-11-25T09:44:38Z

📜 Description

As discussed last week with @markushi, the process will be to make minimal atomic changes in each PR and merge directly to main rather than accumulating on an uber feature branch. This allows for easier review/feedback/corrections, and we can already test a subset of the entire feature "in the field".

The first minimal change is a basic tombstone integration:

exposes internal option to enable/disable (disabled by default)
the integration will only ever be enabled if the runtime system is at least Android 12
the current setup suggests an operation where users will have to disable NDK support (or only depend on sentry-android-core) or get two reports for the same crash
only retrieves the most current tombstone (the current implementation is not entirely correct: we should either remove any remaining ApplicationExitInfo entries with REASON_CRASH_NATIVE or report them too, including the option for the latter; I left this out for minimal interface exposure in the first step, but either variant is easy to add to this PR or later)
adds a basic tombstone parser/decoder and accompanying snapshot test
introduces the following dependencies:
- runtime: protobuf-javalite (the entire features adds ca. 75KiB to the Android sample release APK)
- build: the protobuf plugin and the protoc compiler to automate protocol updates

Open Issues:

While the protobuf runtime is relatively small, there is still the possibility of conflicting with client-side protobuf versions (major versions often introduce quite severe breakage, but I haven't tested this yet, only reviewed change logs).
~~finding clarity in how to proceed with older tombstones (ignore or handling similarly to ANRv2)~~
~~decision if this minimal setup already makes sense to release as an internal API~~
add ManifestMetadataReader to configure conveniently? (or not since the corresponding options are only internal?)
No changes yet to the UI (there are quite a few aspects that would make these stack-traces more readable, from my pov this is currently out of scope, but i am writing stuff down)

💡 Motivation and Context

First sensible release step for #3295

Part of https://linear.app/getsentry/project/tombstone-support-android-0024cb6e3ffd/

💚 How did you test it?

Added a basic parser snapshot test for a serialized tombstone protobuf.
Manual testing.

📝 Checklist

I added GH Issue ID & Linear ID
I added tests to verify the changes.
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
I updated the docs if needed.
I updated the wizard if needed.
Review from the native team if needed.
No breaking change or entry added to the changelog.
No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

Decide if we want to go forward with client-side tombstone processing. This decision should happen with this PR.
- If no, consider sending a tombstone as an attachment, similar to how we attach minidumps and let ingestion/processing deal with decoding (I can prototype this in parallel, since I think we can do most of it in relay, as a first step)
- If yes, decide on adding the protobuf runtime library as a dependency to sentry-android-core (or shade/relocate, or implement our own decoder, given that this is a stable format which only requires a subset of protobuf).
  - Depending on feedback, adaptation to this minimal change (primarily handling older tombstones).
  - Otherwise, integrate an EventProcessor that merges crash events from sentry-native with tombstones.

#skip-changelog (for now)

…ernal)

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

supervacuus · 2025-11-26T21:08:08Z

@markushi, I just realized I cannot omit having a separate "tombstone" marker, even if I report all events, without repeating them. I mean, this was clear to me, but, in addition to it being a must for this PR already, unlike the ANR marker, I must also align it with crashedLastRun.

I wonder if it would make the most sense to do it similarly to ANR by using a TombstoneHint to ensure writing the marker timestamp at the correct life-cycle stage of the event, but introduce a new interface analog to AbnormalExit called CrashExit, so that the hint can take a different route (since it isn't truly an abnormal exit and should affect similar paths to the Native SDK crash marker).

The biggest issue with that approach is that the crashedLastRun is handled entirely in EnvelopeCache rather than AndroidEnvelopeCache, whereas ApplicationExitInfo hint/marker handling happens in the latter.

The PR still makes sense for a first review from you (since, if the general direction makes sense and you have todos not in my list, I can also add a test for the integration itself). Still, I would appreciate a short sync on how to align these execution paths (maybe I don't need to align tombstones with crashedLastRun and can abuse AbnormalExit for the same outcome, though it feels like a significant shortcut, even for minimal internal exposure, which is not at all what I aimed for).

I can convert the PR back to a draft if you prefer.

markushi

Left a few comments - great work so far!

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/proto/io/sentry/android/core/internal/tombstone/tombstone.proto

markushi · 2025-11-27T08:06:57Z

gradle/libs.versions.toml

 otel-semconv = { module = "io.opentelemetry.semconv:opentelemetry-semconv", version.ref = "otelSemanticConventions" }
 otel-semconv-incubating = { module = "io.opentelemetry.semconv:opentelemetry-semconv-incubating", version.ref = "otelSemanticConventionsAlpha" }
 p6spy = { module = "p6spy:p6spy", version = "3.9.1" }
+protobuf-javalite = { module = "com.google.protobuf:protobuf-javalite", version.ref = "protobuf"}


@romtsn Not sure if you followed the conversations, but as you can see here, protobuf requires a runtime dependency. It will have an impact of around 10kb. IMHO fine for now, we should still check how stable this library is to avoid and consumer version mismatch issues.

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry/src/main/java/io/sentry/protocol/SentryStackTrace.java

… S, since that is the earliest version where `REASON_CRASH_NATIVE` provides tombstones via the `traceInputStream`)

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

…rker This currently does not work: While we now can optionally enable reporting of "historical" tombstones, by making the `TombstoneHint` `Backfillable` it will automatically be enriched by the `ANRv2EventProcessor` which is currently the only `BackfillingEventProcessor`. The `ANRv2EventProcessor` is partially written in way that is potentially generic for events with `Backfillable` hints, but other parts are enriching as if those are always were ANRs, which up to now was true, but with Tombstones that assumption now breaks. Next Steps: * There is considerable duplication between the ANRv2Integration and TombstoneIntegration

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

…ns via HistoryDispatcher * also update api dumps and formatting for sentry changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2EventProcessor.java

…tProcessor this handles all events with Backfillable hint, but adds an interface HintEnricher, to allow hint-specific enrichment (like for ANRs) before and after the generic backfilling happened.

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

markushi · 2025-12-03T20:21:34Z

sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java

+        @NotNull SentryEvent event, @NotNull Backfillable hint, @NotNull Object rawHint);
+  }
+
+  private final class AnrHintEnricher implements HintEnricher {


That sounds like a pretty good solution to me, nice!

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneEventProcessor.java

...-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoHistoryDispatcher.java

… iterator instead of for-each

sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry · 2025-12-09T14:47:05Z

sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java

+      if (mainThread == null) {
+        // if there's no main thread in the event threads, we just create a dummy thread so the
+        // exception is properly created as well, but without stacktrace
+        mainThread = new SentryThread();
+        mainThread.setStacktrace(new SentryStackTrace());
+      }
+      event.setExceptions(
+          sentryExceptionFactory.getSentryExceptionsFromThread(mainThread, mechanism, anr));
+    }
  }
-  // endregion
 }


Bug: app.inForeground is missing for TombstoneHint events due to AnrHintEnricher not supporting NativeCrashExit hints.
_{Severity: HIGH | Confidence: High}

🔍 Detailed Analysis

The app.inForeground field is no longer set for TombstoneHint (native crash) events. The new code moved inForeground logic to AnrHintEnricher.setAppForeground(), which only supports AbnormalExit (ANR) hints. As a result, when ApplicationExitInfoEventProcessor.process() handles a TombstoneHint, no enricher is found, and setAppForeground() is never invoked. This is a functional regression, causing missing context data for native crash events that was previously available.

💡 Suggested Fix

Extend AnrHintEnricher or create a dedicated enricher (e.g., TombstoneHintEnricher) to support NativeCrashExit hints and set the app.inForeground field for TombstoneHint events.

🤖 Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java#L665-L782 Potential issue: The `app.inForeground` field is no longer set for `TombstoneHint` (native crash) events. The new code moved `inForeground` logic to `AnrHintEnricher.setAppForeground()`, which only supports `AbnormalExit` (ANR) hints. As a result, when `ApplicationExitInfoEventProcessor.process()` handles a `TombstoneHint`, no enricher is found, and `setAppForeground()` is never invoked. This is a functional regression, causing missing context data for native crash events that was previously available.

_{Did we get this right? 👍 / 👎 to inform future reviews.}
_{Reference ID: 6496195}

That is 100% intentional from my side, at least for now.

cursor · 2025-12-09T14:54:29Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+
+      final SentryStackTrace stacktrace = createStackTrace(threadEntryValue);
+      thread.setStacktrace(stacktrace);
+      if (tombstone.getTid() == threadEntryValue.getId()) {


Bug: Inconsistent thread ID source may misidentify crashed thread

The SentryThread.id is set from threadEntry.getKey() (the map key), but the crashed thread identification on line 77 compares tombstone.getTid() with threadEntryValue.getId() (the Thread's internal id field). If these values ever differ, the crashed thread would not be correctly identified. For consistency and correctness, the comparison at line 77 should use threadEntry.getKey() instead of threadEntryValue.getId(), since that's the source used to set the thread ID.

That is also intentional, because the key and the ID should be equivalent.

supervacuus · 2025-12-09T15:01:22Z

@markushi and @romtsn, I think I also found an okay solution to reduce duplication between the tests for the two AEI integrations. That means the first three "milestones" required for a release (without Native SDK event enrichment/merging) are finished. Please let me know if something is missing from your POV or not yet up to par and how you would like to proceed.

feat: minimal tombstone integration (disabled by default, options int…

ba0c7db

…ernal)

supervacuus requested review from adinauer, lcian, markushi, romtsn and stefanosiano as code owners November 25, 2025 09:44

sentry bot reviewed Nov 25, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

cursor bot reviewed Nov 25, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus commented Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java Outdated Show resolved Hide resolved

redo a seemingly missed spotlessApply

9933e47

cursor bot reviewed Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 2 commits November 26, 2025 19:36

use non-plural option + mark integration also as internal + update API

10b1e94

Merge branch 'main' into feat/tombstone_integration

f12b658

cursor bot reviewed Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java Outdated Show resolved Hide resolved

markushi reviewed Nov 27, 2025

View reviewed changes

init size exceptions since we know we only need one

4b6e1fb

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

sentry bot reviewed Nov 27, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 4 commits December 2, 2025 16:20

move instructionAddressAdjustment to SentryStackTrace

95bd726

add copyright notice to tombstone.proto

4f03857

add historical tombstone option

936a2f0

looks like we need to add a TombstoneEventProcessor anyway.

28652e0

cursor bot reviewed Dec 2, 2025

View reviewed changes

sentry/src/main/java/io/sentry/protocol/SentryStackTrace.java Outdated Show resolved Hide resolved

supervacuus added 2 commits December 2, 2025 16:30

...so let's add it to the options if the SDK supports it (adapt to >=…

920221a

… S, since that is the earliest version where `REASON_CRASH_NATIVE` provides tombstones via the `traceInputStream`)

Adapt AndroidEnvelopeCache to also write Tombstone timestamp markers

48ed5e1

cursor bot reviewed Dec 2, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 4 commits December 2, 2025 16:59

sprinkle with TODOs to highlight next PR steps

6cbfdf8

fix typo

4b1919e

implement TombstoneParser as Closable and close the tombstone stream

d931ddc

eliminate obsolete null check

25f4089

cursor bot reviewed Dec 2, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Show resolved Hide resolved

supervacuus added 3 commits December 3, 2025 09:58

Deduplicate ApplicationExitInfo handling for corresponding Integratio…

7db5895

…ns via HistoryDispatcher * also update api dumps and formatting for sentry changes

update tombstone message construction

a8ea230

Merge branch 'main' into feat/tombstone_integration

92cb264

cursor bot reviewed Dec 3, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Outdated Show resolved Hide resolved

fix abortMessage check

ed81771

cursor bot reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2EventProcessor.java Outdated Show resolved Hide resolved

Merge branch 'main' into feat/tombstone_integration

480bfd6

romtsn mentioned this pull request Dec 3, 2025

Minidump files for native code crashes under Android SDK #4949

Closed

supervacuus added 3 commits December 3, 2025 13:27

Merge branch 'main' into feat/tombstone_integration

c8acdf9

convert AnrV2EventProcessor to a more generic ApplicationExitInfoEven…

125b0c4

…tProcessor this handles all events with Backfillable hint, but adds an interface HintEnricher, to allow hint-specific enrichment (like for ANRs) before and after the generic backfilling happened.

reintroduce, update and correct old inline docs where they make sense.

7e29fbf

cursor bot reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Show resolved Hide resolved

markushi reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneEventProcessor.java Outdated Show resolved Hide resolved

supervacuus added 3 commits December 5, 2025 08:34

remove obsolete TombstoneEventProcessor

605a840

Merge branch 'main' into feat/tombstone_integration

76ef13c

clean up tombstone error handling

7f6dbc9

sentry bot reviewed Dec 5, 2025

View reviewed changes

...-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoHistoryDispatcher.java Show resolved Hide resolved

supervacuus added 2 commits December 5, 2025 09:01

convert ApplicationExitInfoHistoryDispatcher.removeLatest() to use an…

58a342f

… iterator instead of for-each

add TombstoneIntegrationTest

e4b7d78

sentry bot reviewed Dec 9, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java Show resolved Hide resolved

cursor bot reviewed Dec 9, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Outdated Show resolved Hide resolved

supervacuus added 3 commits December 9, 2025 15:31

replace assertions with explicit RuntimeException

ace962f

use test base class for AEI

8cf561e

Merge branch 'main' into feat/tombstone_integration

7edfb15

sentry bot reviewed Dec 9, 2025

View reviewed changes

cursor bot reviewed Dec 9, 2025

View reviewed changes

Uh oh!

feat: minimal tombstone integration #4933

Are you sure you want to change the base?

feat: minimal tombstone integration #4933

Conversation

supervacuus commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

supervacuus commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markushi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markushi Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markushi Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sentry bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

supervacuus Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

cursor bot Dec 9, 2025

Choose a reason for hiding this comment

Bug: Inconsistent thread ID source may misidentify crashed thread

Uh oh!

supervacuus Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

supervacuus commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

supervacuus commented Nov 25, 2025 •

edited

Loading

supervacuus commented Nov 26, 2025 •

edited

Loading

markushi Dec 3, 2025 •

edited

Loading