Skip to content

leafsleuth-ADR-003 - Heuristic Lag Attribution with Confidence Intervals and Web Panel Heatmap-to-Config Remediation Bridge

Status: Accepted Date: 2026-10-27 Formerly: ADR-129 Extends: leafsleuth-ADR-001, leafsleuth-ADR-002, leafconfig-ADR-001, leafconfig-ADR-002, metrics-api-ADR-001, maps-api-ADR-003, leafrtp-ADR-068, leafrtp-ADR-080, leafrtp-ADR-092, leafrtp-ADR-093

Context

When a Minecraft server (Paper, Folia, Purpur, Fabric, NeoForge) experiences tick-time degradation (MSPT spikes or sustained TPS drops), server operators need to answer three operational questions: 1. Where in the world (dimension, region, chunk coordinates) is the lag originating? 2. Who or what build/plugin is driving it, and with what statistical confidence (separating true causal drivers from bystander players who happen to be online at the same time)? 3. Which configuration setting (in the server engine or a third-party plugin) mitigates the bottleneck?

Existing profiling tools leave a gap between low-level JVM stack traces and operator remediation: - CPU / Stack Profilers (spark): Identify which Java method (ServerLevel.tickBlock, EntityTickList.forEach, or a third-party plugin class) consumes CPU time, but do not attribute standard vanilla tick loops to specific spatial coordinates or player bases without heavy per-block instrumentation. - Per-Block / Agent Profilers (Observable): Require client mods and intrusive per-entity/per-block hooks that add profiling overhead and cannot run continuously in production. - Naive Co-Occurrence Counters: Blame whichever player is nearest a chunk or online during a spike without accounting for sample size, variance, or Folia regional tick-thread sharing (bystander exposure).

The monorepo's predict-api, metrics-api, maps-api, and corpus-atlas modules already provide the mathematical and visual primitives needed for continuous, low-overhead probabilistic lag attribution and direct Web Panel config remediation: - predict-api (io.github.dailystruggle.rtp.common.predict): EwmaEstimator (running mean and Welford variance with standardErrorSquared()), EwmaLinearFit (online linear regression $y = \alpha + \beta x$ with standardErrorSquaredAt(x)), BernoulliEstimator (conditional breach probability lift), WorkAttributionLedger<K> (lock-free 4096-entry dual-EWMA 30 s / 10 min cost ledger), and ProbedShareAllocator (Folia split/merge and bystander exposure tracking). - metrics-api: FoliaRegionTpsSampler, PaperMetricsBinding, and regional MSPT/tick-interval telemetry. - maps-api & leafrtp-ADR-080 Web Panel: WebTileRenderer (512x512 PNG tiles with AsyncRenderLimiter bulkhead) for browser-based heatmaps and HeatmapRenderer / SelectionHeatmapRenderer (maps-api-ADR-002) for in-game maps.

Decision

Introduce LeafSleuth, a continuous, zero-bytecode-agent heuristic lag attribution engine built on predict-api and metrics-api, with a primary Clickable Web Panel Heatmap that bridges directly into LeafConfig (leafconfig-ADR-001, leafconfig-ADR-002) for one-click single-server or fleet-wide configuration remediation.

1. Two-Stage Statistical Attribution Engine (predict-api + metrics-api)

To keep continuous runtime overhead near zero (< 0.1 ms/tick and strictly zero synchronous chunk loads per S-005), LeafSleuth separates attribution into a continuous macro-statistical pass and an on-demand micro-probing pass:

A. Continuous Differential Observation (Natural Experiments)

During normal gameplay, players joining, quitting, teleporting, and moving across dimensions naturally activate and deactivate chunk regions (cell groups on Folia per leafrtp-ADR-092, or chunk-ticket clusters on single-loop servers): 1. Marginal MSPT Regression (EwmaLinearFit): - For each active spatial region $r$ and candidate actor $a$, LeafSleuth regresses regional/global $\Delta\text{MSPT}$ against activity transitions (chunk-region load/unload deltas, player presence transitions, and WorkAttributionLedger bursts). - The slope $\hat{\beta}_r$ estimates the region's marginal tick cost (+X.X ms/tick), while $\text{SE}(\hat{\beta}r) = \sqrt{\text{standardErrorSquaredAt}(x)}$ yields a Student-t signal-to-noise ratio and Confidence Interval (% confidence ± margin). 2. Conditional Breach Lift (BernoulliEstimator): - Tracks $P(\text{MSPT} > T)$ relative to baseline breach probability, filtering out global GC pauses or unrelated server-wide spikes that occur regardless of whether region $r$ is loaded. 3. }} \mid \text{Region } r \text{ activeBystander De-Biasing (ProbedShareAllocator & Dual-Window WorkAttributionLedger): - Separates acute burst spikes (30 s half-life EWMA) from chronic background drain (10 min half-life EWMA) and discounts bystander players whose arrival in a shared Folia region did not precede the $\Delta\text{MSPT}$ step-change.

B. Targeted Micro-Probing (Active Confirmation on High-Suspicion Regions Only)

Only when a region's macro-statistical suspicion score crosses a configurable threshold (e.g., estimated impact $\ge 2.0\text{ ms/tick}$ and confidence $\ge 60\%$), LeafSleuth schedules a bounded, budget-paced micro-probe on the region's owning thread (requiring the chunk to remain resident via isChunkLoaded / isOwnedByCurrentRegion; never loading chunks synchronously per S-005): 1. Structural Heuristic Census: Samples already-loaded chunks in the suspect region for tile-entity density (hoppers, pistons, chests, furnaces), entity collision clustering, villager/mob brain density, and redstone/random-tick density. 2. Lightweight Thread-Frame Attribution: During an active tick overrun (leafrtp-ADR-092 interval breach), captures main/region-thread stack frames (ThreadInfo) to identify third-party plugin classes executing inside the hot region. 3. Composite Heuristic Signature Vector: Combines the macro regression (+X.X ± Y.Y ms/tick, confidence %) with the dominant structural or class cause (tile_entity:hopper, entity:villager_ai, entity:collisions, chunk:io_saturation, or plugin:<artifact_id>:<ClassName>). 4. Counterfactual Micro-Dithering (DIAGNOSTIC_MICRO_DITHER), Bounded Taint Flushing (TAINT_FLUSHING), and Contraption Cluster Attribution (clusterMarginalMspt): - When an always-loaded chunk runs at nearly constant load ($\text{Var}(S_{c,t}) \to 0$) so estimate >= 1.5 ms or upperBound95 >= 3.5 ms while lowerBound95 < 1.5 ms, CausalAutoTuner issues a bounded 2-tick deferment probe (DIAGNOSTIC_MICRO_DITHER) out of a 40-tick cycle to collapse the confidence interval in LagAttributionEngine. - EmpiricalFarmBlackBox replaces permanent single-item web poisoning with bounded taintedItemDebt and lastTaintTick quarantine (TAINTED_BY_PLAYER_WEB / TAINT_FLUSHING), amortizing taint debt across verified PROVABLE_GENERATIVE arrivals over a quiet window $T_{\text{clean}}$ without ever inflating yieldPerCausalEvent or minting unverified items. - CausalAutoTuner groups adjacent (Chebyshev distance <= 1) or causally linked (EmpiricalFarmBlackBox#areChunksCausallyLinked) active chunks into a connected contraption cluster (clusterMarginalMspt), preventing multi-chunk contraption dilution across 2x2 or 3x3 chunk boundaries.

2. Primary Web Panel Heatmap & Diagnostic Dossier (WebTileRenderer)

While LeafSleuth supports in-game FILLED_MAP rendering (HeatmapRenderer) and menu-api books for quick checks, the leafrtp-ADR-080 Web Panel is the primary diagnostic surface:

  1. Interactive Fleet & World Heatmap Tab:
  2. Uses WebTileRenderer (maps-api) to stream 512x512 PNG heatmap tiles into the Web Panel, color-coding spatial regions by attributed MSPT impact (+ms/tick) and opacity/border by statistical confidence (%).
  3. In a multi-server network (leafconfig-ADR-002), the operator can switch between hosts (survival-1, survival-2, lobby-1) or view the hottest regions across the entire fleet from one browser tab.
  4. Region Attribution Dossier Drawer:
  5. Clicking any hot tile (rx, rz) on the Web Panel heatmap opens the right-hand contextual drawer displaying:
    • Estimated Impact & Confidence: e.g., +11.4 ± 1.8 ms/tick (88% confidence; 30s burst / 10m sustained).
    • Attributed Actors & Coordinates: Primary associated players (with bystander-corrected confidence weights), dimension, and center block coordinates.
    • Dominant Heuristic Breakdown: Ranked breakdown of structural causes (e.g., Hopper transfer saturation: 64%, Villager AI clustering: 22%) or third-party plugin stack frames (com.example.mobs.WaveTask: 71%).

3. Heatmap-to-Config Remediation Bridge (RemediationBridge SPI)

When both LeafSleuth and LeafConfig (leafconfig-ADR-001, leafconfig-ADR-002) are installed (or within the unified Web Panel), clicking a hot region in the heatmap links the region's Heuristic Signature Vector directly to actionable configuration keys in ast_config_nodes:

  1. Engine & Known-Symptom REMEDIATED_BY Edges:
  2. Built-in heuristic signatures query ast_edges (CAUSES_SYMPTOM / REMEDIATED_BY) across all installed server configs:
    • tile_entity:hopper -> spigot.yml#world-settings.default.ticks-per.hopper-transfer, hopper-check, paper-world-defaults.yml#hopper.cooldown-when-full, hopper.disable-move-event.
    • entity:villager_ai -> paper-world-defaults.yml#entities.behavior.tick-rates.behavior.villager, purpur.yml#world-settings.default.mobs.villager.brain-ticks.
    • entity:collisions -> paper-world-defaults.yml#collisions.max-entity-collisions, entities.spawning.despawn-ranges, bukkit.yml#spawn-limits.monsters.
    • chunk:io_saturation -> paper-global.yml#chunk-system.io-threads, plugins/LeafRTP/config.yml#performance.queueSize.
  3. Third-Party Plugin Bytecode Bridge (READ_BY_CLASS Traversal):
  4. When the micro-prober attributes a tick overrun to a third-party plugin class C (e.g., com.example.mobs.spawner.WaveTask), LeafSleuth traverses the Plane 1 bytecode edge (leafconfig-ADR-001 Section 1): $$\text{Hot Region }(rx, rz) \longrightarrow \text{Class } C \xrightarrow{\texttt{READ_BY_CLASS}} \texttt{ast_config_nodes}(\text{plugin config keys read by } C)$$
  5. Even for an unknown third-party plugin, the Web Panel immediately surfaces the exact YAML keys read by the offending class (e.g., plugins/SomeCustomMobs/config.yml#waves.spawn-interval-ticks).
  6. One-Click Stage & Fleet Sync:
  7. Each recommended key in the heatmap dossier includes a [Mitigate in Web Editor] action that opens the target YAML file at that exact key in LeafConfig's editor pane, stages the adjustment with config-bit-api validation, and (when leafconfig-ADR-002 fleet sync is active) applies the mitigation across all matching servers (role:survival) in one signed commit.

Alternatives Considered

Alternative Why Rejected
Instrument every block/entity tick via a Java agent or Mixin Imposes continuous CPU overhead on the hot path, breaks across loader updates, and conflicts with agentless deployment on shared hosting.
Raw correlation counters without EwmaLinearFit standard error or BernoulliEstimator lift Produces high false-positive blame on bystander players who happen to be online in a hub or shared Folia region when a spike occurs.
In-game chat or book output as the primary heatmap UI In-game maps (128x128) lack the resolution and side-by-side layout needed to inspect multi-server heatmaps, confidence intervals, and linked YAML diffs simultaneously; the Web Panel is primary, with maps-api / menu-api retained as an in-game companion.
Hardcode remediation rules only for Paper and LeafRTP Fails when lag is caused by an arbitrary third-party plugin; traversing leafconfig-ADR-001's READ_BY_CLASS bytecode constant-pool edges generalizes config remediation to any installed plugin JAR.

Consequences

  • Positive:
  • Gives server operators continuous, statistically grounded answers (who and where with % ± SE confidence) without installing client mods or high-overhead bytecode agents.
  • Closes the loop from Detection -> Attribution -> Config Remediation inside a single Web Panel: clicking a lag hotspot jumps straight to the governing Paper, Folia, Purpur, or third-party plugin YAML setting and syncs the fix across the network.
  • Directly reuses predict-api, metrics-api, maps-api, corpus-atlas, and the leafrtp-ADR-080 Web Editor channel.
  • Negative / Trade-offs:
  • Statistical differential observation requires natural activity variance (regions loading/unloading or players moving) over several minutes to narrow the standard error $SE$ on steady-state always-loaded spawn chunks; always-loaded regions rely more heavily on Stage 2 micro-probing.
  • READ_BY_CLASS bytecode linking identifies which config keys a lagging third-party class reads, but when a class reads multiple keys, the operator still chooses which parameter (e.g., interval vs. radius vs. entity cap) to tune in the Web Panel.

References