Skip to content

leafsleuth-ADR-006: In-JVM Passive Forensics, Shared Probe Core, and Host-Aware Discovery Fuzzing


Context

ADR-125 (scripts/bench_fuzz_server.py) introduced whole-server fault and leak discovery across unknown plugin sets using Brigadier command-tree inspection, UCB1 exploration, ddmin shrinking, and jcmd class-histogram screening. Meanwhile, :helpers:stresstest-core provides in-JVM diagnostic probes (LeakProbe, ConsoleWatcher, TicketFootprintProbe, GcSampler, HeapPressureWatcher), and LeafSleuth (leafsleuth-ADR-005) provides live server forensics (ResourceExhaustionForensics).

Three architectural gaps remained between the offline Python harness and the in-JVM LeafSleuth runtime: 1. Bytecode & Packaging Split: :helpers:stresstest-core targeted Java 21 and declared implementation 'org.jacoco:org.jacoco.core:0.8.12', preventing sleuth-common (which targets Java 17 per ADR-113 and must not bundle JaCoCo into LeafSleuth.jar) from linking LeakProbe and ConsoleWatcher directly. 2. Aggregate OldGen Slope Without Class or JAR Blame: ResourceExhaustionForensics regressed post-GC oldGenHeapBaselineMb against chunk tickets (SUSTAINED_HEAP_RATCHET), but could not name which specific Java class or third-party plugin .jar was leaking instances per player departure or command execution on a live server. 3. External-Harness Requirement for Command Auditing & Discovery Fuzzing: Server administrators on managed Paper, Folia, or Pterodactyl hosts could not run passive leak screening (/sleuth audit leaks), Brigadier least-privilege command surface checks (/sleuth audit commands), log/stall signature attribution (/sleuth audit faults), or disposable-gated discovery fuzzing (/sleuth fuzz) directly in-JVM without external Python/RCON drivers or unsafe Bukkit.spigot().restart() calls on Folia.


Decision

1. Shared Probe Core at Java 17 Bytecode Compatibility (REQ-SLEUTH-FUZZ-001)

  • :helpers:stresstest-core compiles with sourceCompatibility = JavaVersion.VERSION_17 and targetCompatibility = JavaVersion.VERSION_17.
  • org.jacoco:org.jacoco.core is scoped as compileOnly in :helpers:stresstest-core (and kept as implementation in :helpers:StressTestRTP, :helpers:stresstest-bukkit, and :helpers:stresstest-fabric), while :resource:LeafFuzz:leaffuzz-common depends on api project(':helpers:stresstest-core').
  • Per leaffuzz-ADR-001, :resource:LeafSleuth:sleuth-common does not depend on :helpers:stresstest-core, keeping LeafSleuth.jar free of fuzzer/harness probes, JMX gcClassHistogram reflection, and plugins/*.jar ZIP scanners (SleuthCommonBoundaryArchTest).

2. In-JVM JarClassOwnershipIndex & Shaded Collision Filtering (REQ-SLEUTH-FUZZ-001)

  • JarClassOwnershipIndex scans plugins/*.jar off-thread via ZipFile (including nested META-INF/jars/*.jar and META-INF/jarjar/*.jar archives), extracting .class entries and plugin descriptors (paper-plugin.yml, plugin.yml, fabric.mod.json, META-INF/neoforge.mods.toml).
  • Classes present in exactly one non-self plugin JAR resolve to that owning .jar via ownerJarForClass(String) and firstSuspectJar(List<String>).
  • Classes present in > 1 JARs (e.g., unrelocated or identically shaded libraries) are recorded in sharedClasses() and excluded from sole-culprit blame so shared dependencies never cause false convictions.

3. Passive Per-Class Retention Screening with GC-Pause Exclusion (REQ-SLEUTH-FUZZ-002)

  • ClassRetentionScreener wraps LeakProbe (DiagnosticCommand gcClassHistogram) and evaluates per-class post-GC instance and byte retention across checkpoints (screenClassGrowth and screenDepartureRetention).
  • To prevent full-GC diagnostic histogram pauses from confounding LagAttributionEngine tick MSPT attribution, LeakProbe#inGcPauseWindow(long) records a +/- 500 ms exclusion window around every gcClassHistogram invocation, which SleuthTickSampler consults before attributing tick duration spikes.
  • ClassRetentionScreener fits ordinary least squares (OLS) slopes per player departure ("departure"), command execution ("command"), or checkpoint ("checkpoint"), applying a Bonferroni family-wise significance threshold ($\alpha = 0.05 / K$ across $K$ candidate classes) and attributing each significant ClassRetentionFinding to its owning .jar via JarClassOwnershipIndex.
  • ResourceExhaustionForensics and ResourceCulpritReport integrate ClassRetentionScreener findings (topRetainedClasses, suspectPluginJar), and /sleuth audit leaks [snapshot|screen] runs snapshots and screening asynchronously via RTPScheduler (S-005).

4. Brigadier Command-Surface Auditing & Fault Signature Attribution (REQ-SLEUTH-FUZZ-003)

  • CommandSurfaceAuditor and PaperCommandTreeInspector inspect the live command graph across default vs op permission tiers using a synthetic PermissibleBase/CommandSender (Command#testPermissionSilent and per-node Brigadier CommandNode#canUse / requires evaluation), build distinct per-tier subtrees, fold namespace:cmd aliases, enforce DEFAULT_DENYLIST, and flag state-mutating commands exposed to unprivileged players (/sleuth audit commands).
  • LogAndStallSignatureOracle combines ConsoleWatcher#attachRootLoggerTap JUL root-logger exception interception (>= Level.WARNING or carrying a Throwable) and a fast 1-tick async in-flight stall watchdog (SleuthTickSampler#sampleInFlightTickStall + LeakProbe#threadPrint for active ticks exceeding 100 ms after subtracting both diagnostic LeakProbe#inGcPauseWindow and ordinary GC collection-time deltas), deduplicating signatures into exc:<ExceptionClass>:<TopPluginFrame> and stall:<TopPluginFrame> attributed via JarClassOwnershipIndex (/sleuth audit faults).

5. Disposable-Gated In-JVM Discovery Fuzzing & Host-Aware RestartStrategy (REQ-SLEUTH-FUZZ-004, REQ-SLEUTH-FUZZ-005)


Consequences

Positive

  • Zero Probe Duplication: StressTestRTP and LeafSleuth share a single Java 17 LeakProbe, ConsoleWatcher, and TicketFootprintProbe implementation in :helpers:stresstest-core.
  • Single-Boot Class & JAR Blame: Administrators on Paper and Folia can run /sleuth audit leaks snapshot and /sleuth audit leaks screen on live servers to identify the exact leaking Java class and owning plugin .jar without restarting the JVM.
  • Strict S-005 & Folia Safety: All JAR ZipFile scanning and gcClassHistogram captures run asynchronously via RTPScheduler off the tick thread, and diagnostic GC pauses are excluded from MSPT lag attribution.

Trade-Offs & Guardrails

  • Diagnostic Full-GC Cost on Snapshot: Capturing a post-GC class histogram (GC.class_histogram) triggers a stop-the-world GC pause; /sleuth audit leaks snapshot is therefore an explicit operator/harness action guarded by an atomic busy lock and the LeakProbe#inGcPauseWindow MSPT exclusion window.