leafsleuth-ADR-006: In-JVM Passive Forensics, Shared Probe Core, and Host-Aware Discovery Fuzzing¶
- Status: Superseded in part by leaffuzz-ADR-001 (in-JVM fuzzer, leak retention, command-surface, and fault-signature probes extracted to
resource/LeafFuzz;SleuthExtensionRegistrySPI retained insleuth-common) - Date: 2026-10-29
- Scope:
helpers/stresstest-core,resource/LeafFuzz, &resource/LeafSleuth(SleuthExtensionRegistrySPI) - Governing ADRs: ADR-113, ADR-115, ADR-122, ADR-125, ADR-129, leafsleuth-ADR-001, leafsleuth-ADR-005, leaffuzz-ADR-001
- Requirements:
REQ-SLEUTH-FUZZ-001,REQ-SLEUTH-FUZZ-002,REQ-SLEUTH-FUZZ-003,REQ-SLEUTH-FUZZ-004,REQ-SLEUTH-FUZZ-005,REQ-SLEUTH-RES-001,S-002,S-004,S-005
Context¶
ADR-125 (scripts/bench_fuzz_server.py) introduced whole-server fault and leak discovery across unknown plugin sets using Brigadier command-tree inspection, UCB1 exploration, ddmin shrinking, and jcmd class-histogram screening. Meanwhile, :helpers:stresstest-core provides in-JVM diagnostic probes (LeakProbe, ConsoleWatcher, TicketFootprintProbe, GcSampler, HeapPressureWatcher), and LeafSleuth (leafsleuth-ADR-005) provides live server forensics (ResourceExhaustionForensics).
Three architectural gaps remained between the offline Python harness and the in-JVM LeafSleuth runtime:
1. Bytecode & Packaging Split: :helpers:stresstest-core targeted Java 21 and declared implementation 'org.jacoco:org.jacoco.core:0.8.12', preventing sleuth-common (which targets Java 17 per ADR-113 and must not bundle JaCoCo into LeafSleuth.jar) from linking LeakProbe and ConsoleWatcher directly.
2. Aggregate OldGen Slope Without Class or JAR Blame: ResourceExhaustionForensics regressed post-GC oldGenHeapBaselineMb against chunk tickets (SUSTAINED_HEAP_RATCHET), but could not name which specific Java class or third-party plugin .jar was leaking instances per player departure or command execution on a live server.
3. External-Harness Requirement for Command Auditing & Discovery Fuzzing: Server administrators on managed Paper, Folia, or Pterodactyl hosts could not run passive leak screening (/sleuth audit leaks), Brigadier least-privilege command surface checks (/sleuth audit commands), log/stall signature attribution (/sleuth audit faults), or disposable-gated discovery fuzzing (/sleuth fuzz) directly in-JVM without external Python/RCON drivers or unsafe Bukkit.spigot().restart() calls on Folia.
Decision¶
1. Shared Probe Core at Java 17 Bytecode Compatibility (REQ-SLEUTH-FUZZ-001)¶
:helpers:stresstest-corecompiles withsourceCompatibility = JavaVersion.VERSION_17andtargetCompatibility = JavaVersion.VERSION_17.org.jacoco:org.jacoco.coreis scoped ascompileOnlyin:helpers:stresstest-core(and kept asimplementationin:helpers:StressTestRTP,:helpers:stresstest-bukkit, and:helpers:stresstest-fabric), while:resource:LeafFuzz:leaffuzz-commondepends onapi project(':helpers:stresstest-core').- Per
leaffuzz-ADR-001,:resource:LeafSleuth:sleuth-commondoes not depend on:helpers:stresstest-core, keepingLeafSleuth.jarfree of fuzzer/harness probes, JMXgcClassHistogramreflection, andplugins/*.jarZIP scanners (SleuthCommonBoundaryArchTest).
2. In-JVM JarClassOwnershipIndex & Shaded Collision Filtering (REQ-SLEUTH-FUZZ-001)¶
JarClassOwnershipIndexscansplugins/*.jaroff-thread viaZipFile(including nestedMETA-INF/jars/*.jarandMETA-INF/jarjar/*.jararchives), extracting.classentries and plugin descriptors (paper-plugin.yml,plugin.yml,fabric.mod.json,META-INF/neoforge.mods.toml).- Classes present in exactly one non-self plugin JAR resolve to that owning
.jarviaownerJarForClass(String)andfirstSuspectJar(List<String>). - Classes present in
> 1JARs (e.g., unrelocated or identically shaded libraries) are recorded insharedClasses()and excluded from sole-culprit blame so shared dependencies never cause false convictions.
3. Passive Per-Class Retention Screening with GC-Pause Exclusion (REQ-SLEUTH-FUZZ-002)¶
ClassRetentionScreenerwrapsLeakProbe(DiagnosticCommandgcClassHistogram) and evaluates per-class post-GC instance and byte retention across checkpoints (screenClassGrowthandscreenDepartureRetention).- To prevent full-GC diagnostic histogram pauses from confounding
LagAttributionEnginetick MSPT attribution,LeakProbe#inGcPauseWindow(long)records a+/- 500 msexclusion window around everygcClassHistograminvocation, whichSleuthTickSamplerconsults before attributing tick duration spikes. ClassRetentionScreenerfits ordinary least squares (OLS) slopes per player departure ("departure"), command execution ("command"), or checkpoint ("checkpoint"), applying a Bonferroni family-wise significance threshold ($\alpha = 0.05 / K$ across $K$ candidate classes) and attributing each significantClassRetentionFindingto its owning.jarviaJarClassOwnershipIndex.ResourceExhaustionForensicsandResourceCulpritReportintegrateClassRetentionScreenerfindings (topRetainedClasses,suspectPluginJar), and/sleuth audit leaks [snapshot|screen]runs snapshots and screening asynchronously viaRTPScheduler(S-005).
4. Brigadier Command-Surface Auditing & Fault Signature Attribution (REQ-SLEUTH-FUZZ-003)¶
CommandSurfaceAuditorandPaperCommandTreeInspectorinspect the live command graph acrossdefaultvsoppermission tiers using a syntheticPermissibleBase/CommandSender(Command#testPermissionSilentand per-node BrigadierCommandNode#canUse/requiresevaluation), build distinct per-tier subtrees, foldnamespace:cmdaliases, enforceDEFAULT_DENYLIST, and flag state-mutating commands exposed to unprivileged players (/sleuth audit commands).LogAndStallSignatureOraclecombinesConsoleWatcher#attachRootLoggerTapJUL root-logger exception interception (>= Level.WARNINGor carrying aThrowable) and a fast 1-tick async in-flight stall watchdog (SleuthTickSampler#sampleInFlightTickStall+LeakProbe#threadPrintfor active ticks exceeding100 msafter subtracting both diagnosticLeakProbe#inGcPauseWindowand ordinary GC collection-time deltas), deduplicating signatures intoexc:<ExceptionClass>:<TopPluginFrame>andstall:<TopPluginFrame>attributed viaJarClassOwnershipIndex(/sleuth audit faults).
5. Disposable-Gated In-JVM Discovery Fuzzing & Host-Aware RestartStrategy (REQ-SLEUTH-FUZZ-004, REQ-SLEUTH-FUZZ-005)¶
- Active fuzzing (
/sleuth fuzz) is gated behindDisposableServerSafetyGate(requiringplugins/LeafSleuth/.bench-disposableandonline-mode=false). SleuthServerFuzzerdrivesNodeBandit(UCB1),NumericBoundaryLadder,FuzzActionShrinker(ddmin+ numeric bisection), andDualAudienceMitigationReport.PaperRestartCoordinatorselectsRestartStrategy.NONEon Folia (ServerAccessor#isFolia()),SPIGOT_RESTARTon Paper whenspigot.ymlsettings.restart-scriptexists, orSUPERVISOR_STOPon Pterodactyl (P_SERVER_UUID), persisting campaign state and.jar.sleuth-disabledrestoration manifests inFuzzCampaignStateStore(plugins/LeafSleuth/fuzz-campaign.yml).
Consequences¶
Positive¶
- Zero Probe Duplication:
StressTestRTPandLeafSleuthshare a single Java 17LeakProbe,ConsoleWatcher, andTicketFootprintProbeimplementation in:helpers:stresstest-core. - Single-Boot Class & JAR Blame: Administrators on Paper and Folia can run
/sleuth audit leaks snapshotand/sleuth audit leaks screenon live servers to identify the exact leaking Java class and owning plugin.jarwithout restarting the JVM. - Strict
S-005& Folia Safety: All JARZipFilescanning andgcClassHistogramcaptures run asynchronously viaRTPScheduleroff the tick thread, and diagnostic GC pauses are excluded from MSPT lag attribution.
Trade-Offs & Guardrails¶
- Diagnostic Full-GC Cost on Snapshot: Capturing a post-GC class histogram (
GC.class_histogram) triggers a stop-the-world GC pause;/sleuth audit leaks snapshotis therefore an explicit operator/harness action guarded by an atomic busy lock and theLeakProbe#inGcPauseWindowMSPT exclusion window.