ADR-125 - Whole-Server Discovery Fuzzing: Protocol Discovery, Agentless Observation, Attribution After Detection¶
Status: Accepted. Amends ADR-111 section 2.2 (Plugin-agnostic inputs, Isolation, Same-run control and Stdlib Python), and refines section 2.3 (Attribution) and section 2.7 (Action vocabulary), for a new whole-server mode only. Also amends stress-test-ADR-001 decision 6 for server mode only (section 2.6). ADR-122 is itself Proposed: the carve-out takes effect only if both are accepted, and if ADR-122 is rejected the profile fields of section 2.6 are defined by this ADR alone. ADR-111 target mode is unchanged. Date: 2026-10-23 Relevant Requirements: REQ-RTP-S-002, REQ-RTP-S-004 Related ADRs: ADR-111, stress-test-ADR-001, ADR-091, leafrtp-ADR-061, ADR-094
1. Context¶
ADR-111 Tier L fuzzes one named target at a time. Its per-target inputs are the command each bot sends (--cmd), optional package prefixes (--plugin-packages) and the target's plugin and jar names (--names, --jars). Each target runs on a server restarted with only its own jar plus the harness (section 2.2 Isolation). That design found a real defect: BetterRTP 3.6.13 retains the sessions of players who quit while a teleport is pending (LEAK_FUZZING.md, run fuzz-20261010-213747). It found it because the operator already knew which plugin to isolate and which command (/betterrtp:rtp) to send. The fit that flagged it scored a player census that named player classes. That census has since been removed in favour of the per-departure class screen, and whether the class screen alone flags this defect has not been measured.
A server admin with forty plugins does not know which plugin to suspect or which command to send. Three things in the current tool assume that knowledge:
- Vocabulary is fixed.
ACTIONSandDEFAULT_CMDSinscripts/bench_fuzz.pyhold a short list of verbs and one namespaced command per known plugin. Section 2.7 planned discovery fromplugin.ymland tab completion, which is Bukkit-only and does not reflect what a given permission level can actually run. - Observation needs the harness. Class histograms and plugin ownership come from StressTestRTP (
LeakProbe), a Bukkit plugin. stress-test-ADR-001 plans Fabric and NeoForge carriers, but they do not exist yet. - Attribution needs package prefixes.
ServerLogOracleattributes frames throughPLUGIN_PACKAGESplus--plugin-packages, and stalls are attributed toServer.
Three properties of a running server make a target-free mode possible:
- The server sends every client a
declare_commandspacket at login: the Brigadier command graph, already filtered by that client's permissions. The packet has the same shape on Paper, Fabric and NeoForge. - Any HotSpot JVM can be observed from the same host with
jcmd <pid> GC.class_histogram(live objects, which forces a full GC) andjcmd <pid> Thread.print, with no agent installed. - The jar set is a list of files. Every class a finding names can be mapped back to the jar that contains it, and a reproduced finding can be bisected over that list by restarting the server.
2. Decision¶
2.1 Two modes¶
- Target mode (
bench_fuzz.py, default): ADR-111 unchanged, including its inputs, isolation,directcontrol and verdict. - Server mode (
bench_fuzz.py --mode server): fuzzes the whole installed jar set. Its only inputs shall be the host, the RCON details, the server directory, a platform profile (2.6), the permission tiers (2.5), the denylist extension (--deny) and the search budget. It shall not take--cmd,--names,--jarsor--plugin-packages.
2.2 Amendment to ADR-111 section 2.2 (server mode only)¶
- Plugin-agnostic becomes target-free: no per-target input at all. No oracle, generator rule, score term or built-in denylist entry shall name a plugin, mod or class. Operator-supplied
--denyentries (2.5) are exempt: they are site policy, not detection logic, and may name plugin commands. - Isolation is relaxed: exploration runs with every installed jar loaded at once. Isolation moves after detection, into the jar
ddmin(2.4), which restarts the server with jar subsets. - Same-run control becomes the vanilla baseline: run first in the same run, on the same server build with the jar folder emptied except for the profile's prerequisite jars (2.6). Its outputs shall never be reused across runs. A signature or class that the baseline also produced shall never be replayed, reported or blamed on a jar.
- Same seeds means the same seed list and generator, applied to the tree the baseline itself discovers. The vanilla tree is a subset of the full tree, so the baseline does not run the programs exploration runs. It is a population control (class-growth fits, tick-gap p99, log signatures), not a per-program control. The per-finding vanilla check is the prerequisites-only replay of 2.4.
- Stdlib Python is widened: in server mode Python may also read
jcmdoutput (text, throughsubprocess) and jar archives through the stdlibzipfilemodule (class entries for the jar index;plugin.yml,paper-plugin.ymlandfabric.mod.jsonfor hard dependencies). It shall still import no third-party package. - Seeded holds unchanged: every episode stays reproducible from its seed and trace.
2.3 Discovery from the protocol¶
- Each bot shall capture the
declare_commandspacket it receives at login and serialise the node graph (kind, name, parser and properties, children, redirect, executable flag, suggestion type) into a swarm event. Adiscovercontrol op shall log in one bot per permission tier for this purpose. - The driver shall merge the trees of all tiers into one
commands.json, recording per node which tiers can see it and its minimum visible tier (defaultbeforeop). Namespaced duplicates (betterrtp:rtpandrtp) shall fold into one node with aliases. Redirects shall resolve with cycle protection. - Argument values shall be generated from the declared parser type and its bounds (numeric ranges,
bool, word / quoted / greedy strings, entity and game-profile selectors resolved to the bot or another bot, block and vector positions near the bot) and from tab-completion replies, cached per node. Numeric parsers (brigadier:integer,brigadier:long,brigadier:float,brigadier:double) with wide or default bounds shall sample from a boundary and magnitude ladder (min,-1,0,1,16,64,256,1024, a clamped upper cap andmax, alongside uniform draws within bounds) so threshold bugs are reached without spending the budget on extreme 32-bit values.plugin.ymland mod metadata shall not be used for discovery. - Bots choose behaviour through a bandit over (command node, follow-up shape) pairs. When both
defaultandoptiers are active, the bandit shall stratify its episode budget sodefault-visible (player-reachable) nodes receive the primary share andop-only nodes cannot starve the player surface;--tiers defaultruns player-reachable nodes only. Follow-up shapes include quitting at once, quitting during a pending delay, rejoining, repeating, acting while another bot runs the same node, and rapid slot clicks plus window close when a command opens an inventory window. Becausebench-swarm.jsdisables Mineflayer'sinventoryplugin under lean mode, window exploration shall handleopen_window,window_clickandclose_windowpackets directly at the protocol level (or enable inventory handling in fuzz mode). Rewards are novelty (a new reply shape or log signature, as ADR-111 section 2.4 keys) and harm (class growth per action, tick-gap excess, kicks).
2.4 Detection first, attribution after¶
- Leak oracle: a per-feature class screen fits each class's post-GC growth against per-episode feature counts of the form
node,node x follow-upand departures. It shall be Bonferroni-corrected over classes and features, and compared against the vanilla baseline. No class name shall be hardcoded (as ADR-122 decision 4 requires of the harness). - Stall oracle: bots report gaps between
update_timepackets (one per 20 ticks) and keep-alive delay. Two signals are scored against the vanilla baseline, excluding intervals that overlap a driver-requested histogram (a full GC), thread dump or restart: - Acute spike: a single
update_timegap more than--stall-ms(default 2000) above the baseline p99. - Sustained tick inflation: episode-wide
update_timeinterval inflation (mean or p95 excess above the baseline distribution duringrepeatandconcurrentarms) that degrades TPS under spam without crossing the single-gap threshold on one call. The driver shall take one or morejcmd <pid> Thread.printdumps while a stall or elevated tick interval is in progress; the signature is the recurring main-thread top frame that is neither JDK nor server code. This replaces ADR-111's "stalls are attributed toServer" in server mode; a stall with no such frame staysserver. - Log oracle:
ServerLogOraclekeeps the ADR-111 parsing and signature rules; in server mode frames are attributed through the jar index instead of package prefixes. - Jar index: classes and frames shall be mapped to jars by scanning the jar folder's archives. A class in no jar is labelled
server. This is a first suspect, not the verdict. - Shrinking and numeric threshold bisection: before jar attribution, a reproduced finding is shrunk over actions and bots (
ddminandshrink_bots). When the shrunk program contains a numeric argument on the ladder, the shrinker shall bisect between a passing bound and the failing value to record the minimal reproducing threshold. - Pre-
ddminprobes:ddmin(scripts/bench_fuzz.py) assumes the full set reproduces, never tests the empty set and always returns at least one item. Two probes shall therefore run first, in order, each a restart followed by--finding-nreplays of the shrunk program: - Full set (every installed jar). Fewer than
--finding-kreproductions: the finding is reported as not reproducible after restart and is not attributed. - Prerequisites only (the empty candidate subset). At least
--finding-kreproductions: the finding is labelledserver(vanilla or a prerequisite), noddminruns and no jar is blamed. A program that sends a node absent from the vanilla tree gets the server's unknown-command reply and counts as not reproducing. - Jar
ddmin: a finding that reproduces (--finding-kof--finding-nreplays, ADR-111 section 2.4), has been shrunk and has passed both pre-ddminprobes shall be confirmed by delta debugging over the non-prerequisite jar set. Each probe restarts the server with a subset plus the prerequisites and replays the shrunk program--finding-ntimes; it passes on at least--finding-kreproductions. Subsets shall be closed under hard dependencies read from jar metadata (depend,depends), and a probe in which a jar of the subset did not load is unresolved and counts as not reproducing. The result is one jar or a minimal set that needs each member. - Verdict and resolution output: ADR-111 section 2.5 holds, with two additions in
verdict.jsonandreport.txt: - Operator mitigation: the canonical command node, all discovered aliases, the minimum reproducing permission tier (
defaultvsop, ranking player-reachable findings first), and any bisected argument bound, so an operator can block, alias-remap or revoke the command incommands.ymlor their permission plugin when removing the jar is not immediate. - Author reproducer: the oracle, the blamed jar set returned by
ddmin(or the jar index's unconfirmed first suspect onINCONCLUSIVE), the leaked class slope (GC.class_histogram) or culprit stack frame (Thread.print/ log), the shrunk program, the reproduction count (kofn) and a one-line--replaycommand.
2.5 Two permission tiers and a denylist¶
--tiers default,op(or--tiers defaultfor player-surface hardening). Default-tier bots join with no extra permissions and inherit whatever the server's existing configuration grants a new player; unlike target mode, server mode shall never runLuckPermsor permission-plugin grants (lp group default permission set ...), preserving the server's real boundary betweendefaultandop. Op-tier bots are opped through RCONop <name>before they join and de-opped withdeopat episode end and at driver exit, including on error. No permission plugin is required.- A denylist entry is a pattern
root [subpath] [<argkind>], matched after alias folding (sominecraft:stopmatchesstop): rootis a root literal with an optional trailing*glob (ban*matchesbanandban-ip). A root-only entry denies the root and its whole subtree.subpathis a sequence of child literals (reload confirmdenies only that path;reloadalone stays allowed).<argkind>denies any path whose next argument node has that parser kind (kill <entity>denieskillwith a selector; barekill, which targets the sender, stays allowed).- Built-in entries:
stop,restart,op,deop,ban*,pardon*,whitelist,save-off,kill <entity>,execute,reload confirm,kick,difficulty,gamerule, and the state-changingticksubcommandstick freeze,tick unfreeze,tick rate,tick sprintandtick step(vanilla 1.20.3+), because they change the tick rate the stall oracle measures.tick querystays allowed. --denyadds operator entries in the same syntax; nothing shrinks the list. A denied node shall never be sent, by generation, mutation, shrinking or replay.- Server mode shall run only against a disposable bench copy made by
scripts/bench_setup.py. No mechanism restores the world,ops.jsonor plugin configs between runs: each run shall start from a fresh copy, and an automatic restore is not part of this decision. The denylist limits damage; the copy bounds it.
2.6 Agentless observation and the platform profile¶
- The driver runs on the server host. It reads histograms and thread dumps with
jcmd; the JVM is found from--pid-fileor fromjcmd -lmatched against the server jar, and for a container the command is wrapped asdocker exec <container> jcmd 1 .... A missingjcmd, wrong user or unmatched PID shall fail fast with a hint; it shall never be read as an empty histogram (ADR-122 decision 2, not measured). - If
/rtpstressanswers over RCON at the start of the run, StressTestRTP checkpoints may replace thejcmdhistogram. The harness is optional in server mode, and the observer is chosen once per run. When the harness is chosen, its jar shall be treated as a prerequisite: kept in the baseline, in both pre-ddminprobes and in everyddminsubset, and never addmincandidate, so every comparison uses the same observer (as ADR-111 section 2.2 kept "plus the harness"). - The per-platform input is a profile (
paper,fabric) extending ADR-122 decision 6: jar folder, log exception patterns, prerequisite jars always kept (for example the Via plugins on Paper, Fabric API and the loader on Fabric) and the restart command. A NeoForge profile follows the same shape. - Carve-out from ADR-122 decision 6 (server mode only). Decision 6 requires that the ADR-111 invariants, isolated included, hold under every profile, and that stack-trace attribution through
--plugin-packagesapply to every profile. In server mode isolated is replaced by the pre-ddminprobes and jarddmin(2.4), and--plugin-packagesattribution by the jar index. Target mode keeps both clauses under every profile.
2.7 Gates¶
- ADR-111 section 2.8 holds: server mode is opt-in and local, never part of
gradlew testor CI, and every oracle needs a planted-bug test against the fake server inscripts/tests/before it can affect the verdict. - The method shall be validated against a planted-bug fixture with neutral names (a Paper plugin and a Fabric mod: a session that leaks when the player quits before a short timer completes, and a radius-gated main-thread busy loop), and against the known BetterRTP 3.6.13 positive. No fixture, BetterRTP or class name shall appear in any input, oracle, rule or built-in denylist entry, and evidence runs shall pass no
--denyentry that names them. - A published claim about another plugin's finding shall follow leafrtp-ADR-061.
3. Alternatives Considered¶
| Alternative | Why Rejected |
|---|---|
Discover commands from plugin.yml / fabric.mod.json (ADR-111 section 2.7) |
Platform-specific, misses Brigadier-registered commands and argument types, and does not reflect what a permission level can run |
| Keep isolation: run target mode once per installed jar | Needs a command per jar the operator does not have, and cannot see interactions between jars; one restart per jar costs more than one exploration plus a ddmin |
| Require the ADR-122 harness on every platform | Blocks Fabric and NeoForge on carriers that do not exist yet; jcmd already gives histograms and thread dumps |
| Attach a JVMTI or Java agent for allocation attribution | Changes launch flags and runtime behaviour; heavier than needed when a class-to-jar index plus ddmin confirms blame |
| Attribute by package prefixes | Needs a per-plugin list, which is the knowledge server mode does not have; shaded libraries share prefixes across jars |
| Permission tiers through LuckPerms groups | Bukkit-only and an extra dependency; RCON op works on every platform |
| Allow-list of safe commands instead of a denylist | Would need per-plugin knowledge and would hide the very commands under test |
4. Consequences¶
Positive¶
- One command fuzzes an unknown Paper or Fabric server and reports "jar X, class Y per
node x follow-up" or "jar X stalls the main thread after node Z" without being told what is installed. - Blame is evidenced twice: by the jar index and by restarting without the jar. A finding that reproduces on the prerequisites alone is never blamed on a jar.
- Discovery reflects the permissions each tier really has, preserves the server's configured
defaultgroup, and stratifies the bandit sodefault-tier (player-reachable) commands are not starved by a largeopcommand tree. - Findings emit both immediate operator mitigation (command node, aliases, minimum reproducing tier and bisected argument threshold) and an author-ready reproducer (
ddminjar set, shrunk program, class slope or stack frame, and--replaycommand).
Negative / Trade-offs¶
- Many plugins add noise. The Bonferroni correction lowers sensitivity, and an intermittent leak such as BetterRTP's (about 24 or more episodes in target mode) may need a larger budget when it shares the bandit with every other node.
- Attribution costs restarts: two pre-
ddminprobes, then for one culprit on the order of 2 x log2(n) subset probes, each a restart plus up to--finding-nreplays. Interactions that need two jars cost more. - Dropping the state-changing
ticksubcommands means a plugin bug reachable only under a frozen or sprinting tick loop is not explored. - Op-tier fuzzing can damage the world or configs. It is bounded by the denylist and the disposable bench copy, not eliminated.
jcmdneeds a JDK and the server's user on the host, ordocker execinto the container.- The vanilla baseline doubles as the control, so baseline and exploration must use the same seeds and server build.
5. References¶
scripts/bench_fuzz.py(gen_program,mutate,Corpus.consider,ddmin,shrink_bots,screen_classes,screen_departures,ServerLogOracle),scripts/bench_overnight.py(Driver.set_jars),scripts/bench_setup.py,devstack/clients/bench-swarm.js(runFuzzProgram)docs/dev/WHOLE_SERVER_FUZZING.md(method, diagrams, validation targets)docs/dev/BEHAVIOUR_FUZZING_PLAN.md(action vocabulary stage)helpers/StressTestRTP/LEAK_FUZZING.md(BetterRTP 3.6.13 finding, runfuzz-20261010-213747)- Zeller and Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE TSE 28(2), 2002 (
ddmin) - Auer, Cesa-Bianchi and Fischer, Finite-time Analysis of the Multiarmed Bandit Problem, Machine Learning 47, 2002 (UCB1)