StressTestRTP¶
A standalone Spigot/Paper/Folia helper plugin that drives a measurable,
repeatable stress test against any /rtp-style command — RTP, RTP-Pro,
or any competitor — and writes ms-precision metrics aligned to the
front-page comparison table.
This plugin lives under helpers/, not under addons/. It
does not depend on rtp-api, rtp-core, or any specific RTP plugin.
Treats the target plugin as a black box: dispatch a configurable
command, listen for PlayerTeleportEvent, record latency.
Why this exists¶
The RTP front-page table compares RTP / RTP-Pro across five performance
columns. The default target-commands set ships the three competitors
worth measuring on modern (1.21+ / 26.x) servers: HuskHomes,
BetterRTP, and EzRTP, all actively maintained, independent RTP
engines. Older entries (AsyRTP, SorekillRTP, AdvancedRTP, JakesRTP,
EssentialsX /tpr) ship commented-out in config.yml: JakesRTP has no
Folia support, EssentialsX's /tpr is self-only and tends to crash
under high call volume, and the rest are effectively unmaintained on
modern versions. Uncomment any of them for a one-off run after verifying
it loads on the target build. (DonutRTP was reviewed
and excluded — it is a GUI front-end that dispatches BetterRTP commands,
not an independent RTP implementation; see PRE_WRITEUP.md section 6.)
| Column | This plugin's measurement |
|---|---|
Cold-start /rtp |
First successful attempt's latency_ms after start |
Warm-queue /rtp |
Median latency_ms across all successful attempts |
| TPS under 10× burst | Minimum sampled TPS during the run |
| MSPT during eval | p95 sampled MSPT during the run |
| Memory footprint | Peak heap-used MB sampled during the run |
The same JAR drives every plugin in the table. Drop the JAR onto a test
server, change target-command in config.yml, run /rtpstress start,
and read the columns straight off the generated summary.txt.
What it does¶
- Registers
/rtpstress(permissionstresstestrtp.admin, default OP). - Samples TPS / MSPT / heap-used on an async timer once per server tick
(50 ms, configurable via
sample-period-ms). A/rtpcompletes in tens of milliseconds, so a coarser 1000 ms cadence would alias past the heap-growth and MSPT spikes the pipeline produces between samples. - On
/rtpstress start [seconds] [concurrency]: - Rolls a fresh CSV at
plugins/StressTestRTP/runs/<timestamp>.csv. - Loops at ~10 Hz: while concurrency slots are open, picks a roster
player, registers a
PlayerTeleportEventexpectation, and dispatches the configured command from the console sender. - Each
PlayerTeleportEvent(causeCOMMAND/PLUGIN/UNKNOWN) for an expecting player closes its attempt withsuccess=true, records distance, and appends a CSV row. - Attempts that don't see an event within
attempt-timeout-msare recorded asTIMEOUTfailures. - On
/rtpstress export: writes a sibling<timestamp>-summary.txtwith the front-page columns prefilled.
Quick start¶
.\gradlew :helpers:StressTestRTP:build
# JAR: helpers/StressTestRTP/build/libs/StressTestRTP-1.0-SNAPSHOT.jar
Drop the JAR into the target server's plugins/ folder alongside the
RTP-style plugin under test. Restart the server. Then in-game:
/rtpstress start 60 4 # 60 s run, concurrency 4 (round-robin across targets)
/rtpstress sequence 60 30 4 # run each target 60 s, 30 s gap, concurrency 4
/rtpstress ramp betterrtp # fixed-rate stages 5..320 TP/s; finds the stress point
/rtpstress status # rolling p50/p95/p99/min-TPS/p95-MSPT/heap
/rtpstress burst 10 # one-shot 10× burst (TPS-burst column)
/rtpstress export # write <timestamp>-summary.txt
/rtpstress reset-cold # run cache-reset commands (config.yml)
/rtpstress stop # end the current run early
# `sequence` is the recommended head-to-head methodology: each
# target-commands entry is exercised in isolation for `perTargetSeconds`,
# then the harness idles for `gapSeconds` so the server (and any pre-warmed
# queue) recovers before the next plugin runs. Defaults come from
# `sequence.per-target-seconds` / `sequence.gap-seconds` in config.yml.
To benchmark multiple co-installed RTP-style plugins head-to-head in a
single run, edit plugins/StressTestRTP/config.yml and list each as a
separate target-commands entry. Bukkit lets you address a specific
plugin's command by prefixing the plugin name (rtp:rtp vs.
betterrtp:rtp), so plugins that all register /rtp no longer collide:
target-commands:
- label: "rtp"
command: "rtp:rtp player={player}"
- label: "betterrtp"
command: "betterrtp:rtp tp {player}"
- label: "essentialsx"
command: "essentials:rtp {player}"
- label: "asyrtp"
command: "asyrtp:wild {player}"
Attempts are dispatched round-robin across the list, and the CSV's
target_label column lets you slice the results per plugin. The summary
(/rtpstress export) prints a per-target breakdown when more than one
label is observed.
RTP / RTP-Pro syntax: RTP's command parser uses
key=valuearguments (seeapi/commands-api/docs/README.md). Bare/rtp <name>is rejected asinvalid command. Always usertp:rtp player={player}for the in-tree plugin; only competitors typically take a positional name.Per-target sender: an entry may set
sender: consoleorsender: playerto override the globaldispatch-as-playerfor that target only. BetterRTP uses its console form (betterrtp:betterrtp player {player},sender: console) because the player-dispatched form produced zero teleports on Paper 26.2.Back-compat: the legacy scalar
target-command: "..."key is still honoured whentarget-commandsis absent (single entry, labeldefault).
/rtpstress reload is not provided on purpose — re-running the
test from scratch on a known-fresh server state is the only way to
guarantee comparable cold-start numbers between competitors.
Roster¶
config.yml roster accepts:
all(default) — every online player except the operator who issued/rtpstress. If only the operator is online, falls back to driving the operator (the "admin-self" mode from the proposal).permission:<node>— every online player with that permission.names:— combined with a YAML listroster-names: [a, b, c].
The harness only drives players that are currently online. To
benchmark with many players, run the offline-mode bot roster
(devstack/clients/bench-swarm.js, see Ramp mode).
Player-churn retention check (leak proof)¶
A heap-used curve cannot separate a leak from GC sawtooth. The leak check counts what survives a full GC instead, the same way for every plugin:
/rtpstress leakcheck <label> # full GC + class histogram -> runs/leak.csv + runs/leak-*.histo.txt
/rtpstress leakcheck status # busy flag, completed count, last result (for RCON polling)
/rtpstress churn all [namePrefix] # kick players; bench-swarm reconnects them as new sessions
leak.csvcarries churn counters (joins, quits, distinct UUIDs) and post-GC heap totals only; it names no class. Class-level retention comes from the.histo.txtfiles, whichscripts/bench_fuzz.pyscreens class by class against the control and against departures.- Same-name churn (default swarm) gives each rejoin a new
Playerobject with the same offline UUID, which exposesPlayer-keyed retention.bench-swarm.js --churn-renamerejoins churn-kicked bots as<name>g<n>(new UUID), which also exposes UUID- and name-keyed retention. Setleak.auto-op-prefix: "bench_"so renamed bots are opped on join and deopped on quit. - Both commands refuse to run during a measured run (
leakcheck ... forceoverrides): the full GC is a stop-the-world pause, and kicks inside a stage would read as plugin timeouts. - The harness drops its own roster
Playerreferences when a run stops and before every checkpoint, so the harness itself cannot be charged as retention.
What a checkpoint records¶
Besides the leak.csv row and the .histo.txt histogram, every
checkpoint appends long-format rows (epoch_ms,label,metric,value) to
runs/leak-metrics.csv, so any metric can be fitted without knowing the
plugin:
- every memory pool and buffer pool after the full GC, loaded/unloaded classes;
- RSS, peak RSS, swap, OS threads (
/proc/self/status), open descriptors (/proc/self/fd); Linux only, absent elsewhere; - native memory by category (
nmt.*) when the JVM runs with-XX:NativeMemoryTracking=summary; otherwisenmt_enabled=0(NOT MEASURED). The raw summary is kept as.nmt.txt; - thread counts and
threadgroup.<name with digits folded>.<STATE>counts; the full thread dump is kept as.threads.txt; - per plugin: pending
BukkitSchedulertasks (not on Folia:plugin_tasks_measured=0), registered listeners, and live instances / MB of classes shipped in the plugin's own jar (owned_classes; bundled libraries count, a class shipped by several jars counts for none; main package as fallback when the jar is unreadable,owned_classes=-1); rss_minus_nmt_mb: process memory the JVM does not account for (native-library malloc, allocator fragmentation);- workload counters: teleports, chunk loads, generated chunks, joins, quits, uptime (the x-axes for slopes).
/rtpstress leakcheck <label> dump also writes a live-objects
.hprof (a second multi-second pause, about the size of the live heap).
/rtpstress leakcheck <label> quick keeps the full GC, the histogram
file and the leak.csv row, but writes a reduced metrics row: heap after
GC, histogram_ms, workload counters, /proc RSS/threads/open
descriptors, JVM thread counts and the per-plugin census
(checkpoint_quick=1). It skips NMT, memory/buffer pools, the thread
census and both side files (.nmt.txt, .threads.txt), so its cost is
about one full GC; meant for short per-episode deltas.
Controls¶
none: no RTP plugin jar, bots idle.direct: no RTP plugin jar, the harness teleports bots at the soak rate with/rtpstress tp <player>(uniform point withinleak.direct-radius, async chunk load, highest block; Paper/Folia only). Uncomment thedirectentry intarget-commandsto use it. It subtracts world, generation and landing growth that any RTP plugin causes, so it is the preferred control.
Driver¶
scripts/bench_leakcheck.py automates the protocol over RCON: one
restart per target with only that plugin's jar enabled (controls: none
enabled, verified via plugins).
--mode soak(default): c00 checkpoint, then flat ramp segments at--rateTP/s for--segment-s(keep it at or belowspark.timeout-seconds) for--soak-hours, a churn every--churn-everysegments and a checkpoint every--checkpoint-every, then a final checkpoint.--dumptakes a heap dump at the first and last checkpoint.--mode cycles: the earlier N x (ramp, churn, rejoin, checkpoint).
The analysis fits a slope per metric against elapsed hours, subtracts
the control's slope for the same metric, and flags metrics whose excess
is beyond 3 standard errors, rising, and at least 2 % of the start value.
It also lists classes by live-byte growth between the first and last
histogram, net of the control's growth per hour (G1 filler objects
excluded). --analyze-only <stamp> re-reads the server's runs/;
--from-checkpoint 1 drops a c00 that was not a cold baseline.
scripts/heap_owner.py runs Eclipse MAT headless (installed separately)
on the two dumps: leak suspects by snapshot comparison against c00, and
leak suspects on the final dump. It names as owner the first class in
each suspect that a plugin's jar ships (same rule as owned_classes,
plugin.yml main package as fallback), or server/JDK when none does.
Behaviour fuzzing¶
Goals, methodology, how to read a run, and the leaks found so far:
LEAK_FUZZING.md.
scripts/bench_fuzz.py looks for the player behaviour that leaks,
without knowing the plugin. It starts devstack/clients/bench-swarm.js
--fuzz-control <port> itself and runs short seeded episodes: 2-8 bots
each run a program of rtp, wait, move, quit, rejoin (same or new
name), die and rtp2, then a leakcheck <label> quick gives the exact
post-GC count deltas and every class's instance delta. The only
per-target input is the command (direct maps /rtp to /rtpstress tp
{self}); bots get permissions from LuckPerms group default (* or
--grant), reverted on exit. direct runs first and is the control every
other target subtracts.
- Repetitions (
--reps, default 3): each episode's programs runrepstimes with fresh bot names and a checkpoint after each;slope.<metric>(live bytes, heap after GC, RSS, metaspace, loaded classes, buffers, threads, fds, per-plugin tasks and listeners) is the growth per repetition, so a cache that fills once scores about 0. - Console actions (
--console "rtp=rtp reload,...", opt-in): allow-listed RCON commands fired at random times during about 30 % of episodes (--console-p,--console-max), carried through replay, shrink and the ladder. - Prediction: a ridge fit per count against behaviour features
(action counts, quits by time since the last
/rtp, moves during a pending teleport). A feature is flagged when its interval excludes 0 and the control's does not. Classes enter through a cumulative screen net of the control, with a Bonferroni-corrected z. - Learning: a cross-entropy method moves the generator towards the
highest net leak per minute. While nothing is confirmed the search
stays
searching: more novelty per stalled generation (up to--novelty-max), restarts from fresh random points. - Surprises: an episode that grows a count far beyond the control is
replayed at once (
--surprise-n,--surprise-k); a reproduction makes a candidate and switches toexploiting. Candidates are shrunk to the fewest actions that still leak. - Findings: every episode also runs the finding oracles (server log:
uncaught exceptions with a stack trace,
Could not pass event, tick stalls; permission denials). A finding has a stable signature (class, owning plugin, root exception type, first frame in that plugin, digits dropped). A new signature attributed to the target (--names/--jars, plus--plugin-packages) and absent from the control is replayed the first time it appears (--finding-n,--finding-k) and shrunk (--finding-shrink). - Ladder and verdict: candidates climb
--rungs(default4x1,12x3,48x10bots x reps) by successive halving. At the top rung a target fails when the count grows by at least half the prediction and heap after GC beyond the control's; heap dumps are taken andheap_owner.pyruns with--mat. A reproduced target finding also fails, with no ladder (confirmed_bynames the oracle). PASS means nothing found within budget, with the coverage reached; INCONCLUSIVE is a signal that did not confirm (a candidate lost at scale is micro-only; a target finding that did not reproduce is a one-off).
python3 scripts/bench_fuzz.py --server-dir /mnt/threadripper-dir/26.2 --host <bench> \
--targets direct,betterrtp,rtp --seeds 1 --episodes 80 [--mat ~/mat/ParseHeapDump.sh]
Output goes to scripts/tmp/runs/fuzz-<stamp>/: report.txt, and per
target episodes.jsonl (one line per episode, replayable with --replay
FILE --ep K), classes.jsonl, optimiser.jsonl, fit.json,
surprises.jsonl, shrunk.jsonl, ladder.jsonl, findings.jsonl,
finding-replays.jsonl and verdict.json; the control's signatures are
in control-findings.json.
Stop any passive swarm first: the script refuses to run with non-fuzz
players online unless --allow-online is set.
Ramp mode (stress point)¶
start / sequence are closed-loop: one attempt in flight per player
and at most default-concurrency in flight overall. By Little's law the
offered rate is then at most in-flight / latency, so 4 accounts against
a ~300 ms plugin can never offer more than ~13 TP/s, and a fast plugin's
throughput reads as the harness's ceiling, not its own. Ramp mode offers
load open-loop at a fixed rate and finds the rate at which each
plugin stops keeping up.
/rtpstress ramp <target> [stageSeconds] # target = a target-commands label
/rtpstress ramp <target> 120 100 # flat 100 TP/s for 120 s (rates override ramp.stages)
/rtpstress status # current stage + stress point so far
/rtpstress stop # ends the ramp; completed stages are on disk
- Idle window (
ramp.idle-seconds, default 30) with bots online and no dispatch. Its second half is the idle baseline (CPU cores, MSPT p50) applied to every stage, so thenet_*cost-per-teleport columns in<stamp>-phases.csvare filled. It runs before any teleport so a plugin's post-teleport background work (queue fill, generation) is billed to that plugin, not subtracted as ambient. - Optional warm-up (
ramp.warmup-seconds, default 30) at the first stage's rate; no CSV rows are written for it. - Each stage (
ramp.stages, default5,10,20,40,80,160,320TP/s, eachramp.stage-seconds, default 60) issues dispatch slots at exactly the offered rate. Every slot goes to an idle roster player (no attempt in flight, pastramp.per-player-gap-ms, default 0). If none is idle the slot is counted asharness_shed, harness-side saturation, and is not retried. - Concurrency is bounded only by the roster size: still one in-flight
attempt per player, and
default-concurrencyis ignored. - After the dispatch window the stage drains (at most
attempt-timeout-ms+ 1 s), so late completions count toward the stage that dispatched them. Then one row is appended to<stamp>-ramp.csvand flushed before the next stage starts, so a server crash keeps every completed stage.ramp.gap-seconds(default 5) idles between stages. - Each stage is also a normal measurement phase (label
<target>@ramp<i>-<rate>tps), so<stamp>-phases.csvkeeps its CPU, chunk and GC columns per stage. One spark profile spans the ramp.
Stress point¶
A stage passes iff all of these hold (thresholds under ramp.pass):
| Criterion | Default | CSV fail_criteria token |
|---|---|---|
achieved_tps >= 0.95 x offered_tps (achieved = successful teleports / stage seconds) |
min-achieved-fraction: 0.95 |
ACHIEVED_BELOW_OFFERED (+ HARNESS_SATURATED when shed slots alone exceed the 5 % shortfall) |
| MSPT p95 over the stage <= 50 ms | max-mspt-p95: 50.0 |
MSPT_P95 |
(timeouts + errors) <= 1 % of attempts |
max-fail-fraction: 0.01 |
FAILURE_RATE |
The stress point is the highest passing stage. The ramp stops after
ramp.stop.consecutive-failures (default 2) failing stages in a row; a
pass after a single failure resets the count. It also stops if wall-clock
TPS stays below ramp.stop.low-tps-threshold (default 5) for
ramp.stop.low-tps-seconds (default 10) continuously. That stage is
aborted, recorded with aborted=true, and fails LOW_TPS_ABORT. The
end of the ramp logs one summary line and writes <stamp>-ramp-summary.txt:
stress point stage and offered rate, plus the first failing stage and
the criterion it failed on.
A stress point whose next stage failed with HARNESS_SATURATED is a
lower bound: the roster, not the plugin, ran out. Add bots and re-run.
Busy vs failure. Plugin-side "busy / cooldown / already teleporting"
rejections are shed load (busy_rejections): they lower achieved_tps
but are not failures. They are recognised only when the rejection reaches
the console (ConsoleWatcher -> CONSOLE_FAIL: reason) and matches
ramp.busy-patterns. A rejection sent only to the player via chat never
reaches the server log. It surfaces as a timeout, and therefore as a
failure. bench-swarm.js counts such chat replies out-of-band
(busy_chat= in its stats line) so the two can be reconciled.
MSPT is the sampler's value (Paper getAverageTickTime, a ~100-tick
average; wall tick time on Spigot/Folia). TPS for the abort rule and
tps_min is measured by a 20-tick wall-clock timer with a staleness
bound, not Paper's 1-minute getTPS()[0], which lags a 10 s rule.
<stamp>-ramp.csv columns¶
target,stage_index,offered_tps,stage_seconds,attempts,successes,timeouts,errors,
busy_rejections,harness_shed,achieved_tps,achieved_fraction,fail_fraction,
latency_p50_ms,latency_p95_ms,latency_p99_ms,mspt_p50,mspt_p95,mspt_samples,
tps_min,heap_peak_mb,roster_size,peak_in_flight,aborted,complete,verdict,fail_criteria,
start_epoch_ms,end_epoch_ms,
repeating_scheduled,repeating_cancelled,repeating_live_start,repeating_live_end,
repeating_cancel_failures
repeating_* count the harness's own per-player repeating tasks (the Folia
pinned-position watch): scheduled and cancelled during the stage, the live
count when the stage began (after the settle gap; ~0 expected, a warning is
logged above ramp.repeating-task-tolerance, default 2) and when it ended,
and cancel() failures. A non-zero repeating_cancel_failures means a
scheduler task could not be cancelled; the task then stops itself through its
stop flag and a hard tick bound (folia.pinned-position-watch-max-ticks,
default attempt timeout + 2 s). The summary file adds
repeating_live_between_stages_max, repeating_live_at_end and
repeating_cancel_failures.
verdict is PASS, FAIL, or PARTIAL. PARTIAL (complete=false)
marks a stage cut short by /rtpstress stop; it is kept for the record
but never counts toward the stress point. -1 means NOT MEASURED.
Bot roster: bench-swarm.js¶
Full clients cost too much RAM, so the roster is many lightweight
offline-mode Mineflayer bots, split across a few Node worker processes
(--workers, default one per 16 bots, at most CPUs - 1):
cd devstack/clients
npm install
node bench-swarm.js --host <bench-host> --port 25565 --count 64
- Server prerequisites:
online-mode=false(the Linux bench server already runs offline) androster: "all". The bots are namedbench_000..bench_063(--prefix,--start-index). - Connection throttle: every bot connects from one IP, and Paper
rejects reconnects faster than
bukkit.ymlsettings.connection-throttle(default 4000 ms). Either set it to-1for the bench, or keep--join-delay-msabove it (default 4500; 64 bots are then online after ~5 min). All connects and reconnects go through one queue spaced by this delay. - Bots stay idle: no chat, no movement. They auto-respawn on death and
reconnect after a kick or disconnect with exponential backoff
(
--reconnect-base-ms5000 doubling to--reconnect-max-ms60000). Ctrl+C disconnects them cleanly. - Defaults keep memory low:
--view-distance 2(the server still sends real chunks, but only min(client, server) of them; raise it to model real clients' send cost), physics off (no gravity simulation; teleport-confirm and position packets are still sent; use--physicsifallow-flight=falsecauses mid-air fly kicks), and a lean Mineflayer plugin set (--full-pluginsrestores all). - Chunk decode off: each teleport makes the server send a fresh
chunk square, and decoding those on one Node thread saturated the
event loop at ramp rates. Keepalives then queued behind chunk data and
both sides dropped the bots ("client timed out after 60000 ms",
server "keepalive timeout"). The bots now receive chunks and still
acknowledge every chunk batch, so server send cost stays real, but do
not decode them (
--keep-worldrestores decoding). - Server-side load distance: Paper loads chunks around a player by
the server's view/simulation distance, whatever the client asks.
During a ramp the harness caps each roster player to
ramp.player-view-distance(default 2, Paper only, 0 disables), the same for every target. Without it, 64 bots at view-distance 10 meant ~529 fresh chunks per landing: TPS fell below 1 and every attempt timed out before any plugin was measured. - A stats line every 30 s (one per worker) reports online count, connects, kicks (top reasons), errors, deaths, busy chat replies and process RSS.
- With
dispatch-as-player: truethe bots need each target's/rtppermission, and per-player plugin cooldowns must be zero (or bypassed) for the bench. Otherwise every repeat dispatch is a cooldown rejection, not load.
Ramp procedure: start the swarm, wait until online=64/64, run
/rtpstress ramp <target> from the console, then repeat for each target
with a server restart between targets so each plugin starts cold.
Folia¶
Supported. The harness is structured to never violate Folia threading:
- Sampler runs on the async scheduler.
- Runner ticks on the async scheduler.
- Every command dispatch hops to the target player's
EntitySchedulerfirst. - The
PlayerTeleportEventlistener is read-only atMONITORpriority.
Folia detection is automatic via reflective probing of
io.papermc.paper.threadedregions.RegionizedServer. The same JAR drops
onto Spigot, Paper, and Folia unchanged.
Folia matrix (one plugin at a time)¶
scripts/bench_matrix.py runs every Folia-capable target in turn, restarting the server
with only that target's jar enabled (direct = no RTP jar), and writes one table per run.
n=1 per target per pass on one rig: compare rows only within a run. Defaults (folia
profile): a paced ramp 5@820 (about 4,096 teleports), then a ladder 10,20,40@60;
jakesrtp gets one load attempt and is recorded unsupported-not-loaded if Folia refuses it.
# 1. server up under a relaunch loop (Folia has no /restart; the matrix sends `stop` and the loop
# starts the JVM again): copy scripts/folia-start-loop.sh to the server dir as start-loop.sh and run
# `tmux new -s folia 'cd <server-dir> && ./start-loop.sh'` (detach Ctrl-b d; Ctrl-C ends it, or
# `touch .no-relaunch` then `stop`). Then the swarm with byte counters
# (the Folia dir uses game port 25677 / RCON 25685 so it can run beside the Paper dir)
node devstack/clients/bench-swarm.js --host <server> --port 25677 --count 48 \
--stats-file /tmp/swarm-stats.jsonl --swap-stuck --churn-rename
# 2. plan and time estimate, no RCON traffic
python scripts/bench_matrix.py --server-dir <server-dir> --host <server> \
--swarm-stats /tmp/swarm-stats.jsonl --dry-run
# 3. smoke run, then the full list (resume a stopped run with --resume <out-dir>)
python scripts/bench_matrix.py ... --targets direct,rtp --round 5@60
python scripts/bench_matrix.py ...
Output in scripts/tmp/matrix-<stamp>/: matrix.md and matrix.csv (long form),
manifest.csv (outcome, wall window, run stamp, config hashes, swaps, live log findings),
bandwidth.csv, copies of each run's ramp files and the session log. Rebuild the tables
with python scripts/bench_matrix_report.py <out-dir>.
- Bandwidth is wire bytes received by the bots (
socket.bytesReadsummed over bots, compressed), sampled on the driver's clock. Net = stage bytes minus the idle rate (second half of the ramp's idle window) times stage seconds, divided by successful teleports. Stage windows map the server clock to the driver clock with one offset per run (first stage startsidle-seconds + warmup-secondsafter the ramp command). A stalled stats file printsNOT MEASURED, never zero. - Stuck bots.
--swap-stuckreplaces a bot the server stopped teleporting while it keeps teleporting the others (a plugin that locks a player out after an error, as EzRTP did on Folia in20261008-052141): it quits and rejoins under a new name, which is a new offline UUID. The swap count is recorded per run, and each swap adds one join chunk burst to that stage's bandwidth. The harness refuses/rtpstress churnmid-run, so the swarm does it. - Log findings. A live watcher records the first plugin problem and the watchdog and
async-chunk counts per run; the report scans the session log for
Tick region ... has not responded,Cannot retrieve chunk asynchronously, off-thread access errors and exceptions attributed by package. A failure is reported as "failed on this rig, this config" with its first log line. - Latency maximum is the slowest successful teleport dispatched inside the stage window,
read from the harness's per-teleport
<stamp>.csv(latency_ms, which the run copies next to the ramp files). Timed-out attempts are kept out of it and reported separately aslatency_max_failed_attempt_msinmatrix.csv. - MSPT on Folia. The harness's MSPT is a 50 ms timer on the global region, so its p95 reads
about 50.0 whatever the load. Set
ramp.pass.max-mspt-p95: 1000in the Folia config so the ladder is not stopped by it. The table labels the column "global timer"; read the regional TPS columns for tick health. - Profiling (
--spark) needsspark.enabled: trueand a Folia thread selector inplugins/StressTestRTP/config.yml. Folia ticks onFolia Region Scheduler Thread #Nand has noServer thread, so usethreads: "*Region Scheduler Thread*,RTP-Anvil-IO-*"and atimeout-secondslonger than the longest stage (900 for the 820 s ramp). After a short run,python scripts/folia_spark_check.py --server-dir <server-dir> --since-min 15decodes the new profiles and fails when they hold no region-thread samples. Setthreads: "*"once to see every thread name if it does.
CSV format¶
Header (always written, one line):
attempt_id,player,world,target_label,dispatch_epoch_ms,teleport_epoch_ms,latency_ms,
success,fail_reason,from_x,from_z,to_x,to_z,distance,
tps_at_dispatch,mspt_at_dispatch,heap_used_mb_at_dispatch
target_label is the label field of the target-commands entry that
dispatched this attempt (or default when the legacy scalar key is in
use). Filter the CSV by this column to compare plugins.
fail_reason is TIMEOUT, NULL_TO, or empty (success). latency_ms
is -1 for in-flight rows that the server crashed mid-write — anything
else is a wall-clock millisecond delta from dispatch to teleport event.
Front-page summary¶
/rtpstress export writes a human-readable summary that mirrors the
front-page comparison table:
StressTestRTP run summary
generated: 2026-05-01T17:42:13
target-commands: rtp=`rtp:rtp player={player}`, betterrtp=`betterrtp:rtp tp {player}`
attempts: 184 (ok=181)
--- Front-page comparison columns (all targets, combined) ---
Cold-start /rtp: 412 ms
Warm-queue /rtp: 38 ms (median)
TPS under burst: 19.84 (min observed)
MSPT during eval: 12.40 ms (p95)
Memory footprint: 612 MB peak
--- Latency percentiles (success only, all targets) ---
p50: 38 ms
p95: 122 ms
p99: 264 ms
--- Per-target breakdown (success only) ---
rtp n=92 cold=412 ms p50=38 ms p95=84 ms p99=164 ms
betterrtp n=89 cold=701 ms p50=141 ms p95=410 ms p99=812 ms
raw csv: 20260501-174118.csv
Phase-aggregate CPU per teleport¶
Alongside the per-attempt CSV, each run also writes a sibling
<stamp>-phases.csv with one row per measurement phase (one TIMED
window, one BURST, or one SEQUENCE per-target window). Header:
phase_label,start_epoch_ms,end_epoch_ms,wall_ms,attempts,successes,
process_cpu_ms,main_thread_cpu_ms,
cpu_ms_per_attempt_total,cpu_ms_per_attempt_main
process_cpu_ms— total user+system CPU charged to the JVM during the phase. Includes background work (GC, async chunk loaders, other plugins still ticking), so it answers "what does this plugin cost the box", not "what does this plugin alone consume".main_thread_cpu_ms— the server tick thread's CPU time only. This is the most differentiating axis: a plugin that does sync chunk I/O on the tick thread shows huge main-thread CPU; a properly async plugin shows very little. Note that wall-clock blocking (.join()waiting on a chunk future) is not counted as CPU — that gap surfaces in MSPT instead, and the disparity between the two is itself a useful diagnostic.- The two
cpu_ms_per_attempt_*columns are pre-divided convenience values;plot_stress.pyre-derives attempt-weighted means across multiple runs from the rawprocess_cpu_ms/main_thread_cpu_mssums when summarising.
Per-attempt CPU is intentionally not collected: the work for one
/rtp is split across the tick thread, the async chunk loader, the
safety scanner, and the entity scheduler, so per-attempt CPU cannot be
honestly assembled on Bukkit. Phase-aggregate CPU divided by completed
attempts is the most defensible per-teleport number.
plot_stress.py reads the phases CSV automatically when present and
emits two extra charts (cpu_per_tp_total.png, cpu_per_tp_main.png)
plus two extra columns in summary.md. If the phases CSV is missing
(older runs, or if you only kept the per-attempt CSV) those artefacts
are simply omitted — no breakage.
Ticket-footprint calibration (setup phase)¶
Every retained-memory figure the harness reports is quoted per cached
location, and a cached location is one addPluginChunkTicket call.
Turning that into bytes needs one more number: how many chunks the
server actually makes resident in response to a single ticket.
That number is a platform decision, not a plugin one. Vanilla
propagates ticket levels outward, so a ticket below the FULL threshold
pins a neighbourhood rather than one chunk. Paper and Folia each
reimplemented that subsystem and need not agree with vanilla or with
each other. Assume it and every bytes-per-cached-location inference is
off by exactly the factor assumed - in the flattering direction if you
guess low.
So it is measured, once, at plugin enable, before any teleport is recorded:
- Pick a chunk ~20k blocks from origin (well outside the run's
origin-centred teleport radius, so the probe never warms ground the
run then measures) that reports
isChunkLoaded() == false. - Apply exactly one
addPluginChunkTicketon the thread owning that chunk, wait for that chunk to actually report loaded (up to 400 ticks - on a cold start it may need generation, and on Folia a fresh region thread), then countChunkLoadEvents within 7 chunks of it for the settle window. A chunk that never loads is reported as NOT MEASURED, not as a zero footprint. - Release the ticket and count the unloads.
If the enable-time probe raced server start-up, it is retried
automatically at the next /rtpstress start, or on demand with
/rtpstress probe-footprint while no run is active.
Used heap and committed heap are sampled at all three boundaries - before the ticket, after the load settles, after the release settles - and collections inside the window are counted, so the reading covers the whole lifecycle instead of only the growth half.
Counting events rather than diffing World#getLoadedChunks() is
deliberate: the array form allocates a reference to every loaded chunk
on the server, and it is not region-safe to walk on Folia. Events are
already delivered on the owning thread, so the probe reports the same
quantity on Spigot, Paper, and Folia.
Results land in ticket-footprint.txt and in these phase columns:
| Column | Meaning |
|---|---|
ticket_footprint_chunks |
chunks made resident by one ticket |
ticket_footprint_shape |
1x1 / 3x3 / 5x5, or IRREGULAR / NONE |
ticket_footprint_released |
chunks unloaded when the ticket was removed |
ticket_probe_noise_loads |
loads outside the attribution radius |
ticket_footprint_heap_bytes |
used-heap delta across the window |
ticket_footprint_bytes_per_chunk |
that delta per chunk |
ticket_footprint_heap_label |
UNCOLLECTED_ALLOCATION_INCLUSIVE, or UNATTRIBUTABLE_CONCURRENT_ALLOCATION when growth exceeds 16 MiB per counted chunk (someone else allocated in the window) |
ticket_footprint_heap_used_before_bytes |
used heap before the ticket |
ticket_footprint_heap_used_after_load_bytes |
used heap once the load settled |
ticket_footprint_heap_used_after_unload_bytes |
used heap once the release settled |
ticket_footprint_heap_retained_after_unload_bytes |
growth still resident after release |
ticket_footprint_heap_reclaimed_bytes |
growth that came back on release |
ticket_footprint_committed_delta_bytes |
committed-heap change across the probe |
ticket_probe_gc_collections |
collections inside the probe window |
ticket_footprint_reclaim_label |
RECLAIMED_ON_UNLOAD / RETAINED_PENDING_COLLECTION / NO_NET_GROWTH / GC_DURING_WINDOW_UNATTRIBUTABLE |
Reading them:
ticket_footprint_chunksis the multiplier. Multiply by the cached-location cap under test to get the resident-chunk cost of a full hot cache.ticket_probe_noise_loadsdecides whether you can trust it.0means the window was quiet and the footprint is attributable to the ticket. Non-zero means unrelated chunk traffic overlapped the window, and the footprint is an upper bound. Probe an empty server.ticket_footprint_releasedis the retention check. Equal toticket_footprint_chunksmeans retention is bounded and symmetric. A shortfall means the ticket did not fully release, and every residency figure in the run should be read as accumulating.- The heap figures are upper bounds, not retained sets. No
collection is forced - a
System.gc()on a server under measurement would corrupt the GC columns recorded in the same run - so both include transient allocation. The label says so in every row. ticket_probe_gc_collectionsgates the heap columns.0means no collection ran inside the window, so the growth and the post-release reading are attributable to the ticket. Non-zero sets the reclaim label toGC_DURING_WINDOW_UNATTRIBUTABLEand no heap figure from that probe should be quoted.ticket_footprint_reclaim_labelis the OOM-exposure reading, and the interesting one. An unload drops references; it does not free bytes.RETAINED_PENDING_COLLECTION- growth still resident after the ticket is gone, with no collection in the window - is the normal Paper result and is not a leak, it is deferred reclamation. It is also exactly why sizing a heap from steady-state footprint fails: a burst that demands memory faster than the collector reclaims released chunks can exhaust a heap the footprint figure called ample. Size against headroom, not footprint.RECLAIMED_ON_UNLOADmeans most growth came back unaided;NO_NET_GROWTHmeans there was nothing to reclaim.ticket_footprint_committed_delta_bytesinvalidates absolute heap numbers when non-zero. The JVM grew the heap for one ticket, so the run is describing heap-growth policy as much as the plugin. Pin-Xms == -Xmxbefore quoting any absolute figure.
-1 and empty mean NOT MEASURED. In particular, -1 here never means
a one-chunk footprint.
Tunable under ticket-footprint-probe in config.yml
(enabled, origin-distance-blocks, settle-ticks). Raise
settle-ticks on a slow disk if the count looks truncated.
Folia: what completed the row, and which TPS you are reading¶
Two columns exist because Folia's numbers are otherwise easy to misread.
attribution_source (per attempt) names the observation that set
teleport_epoch_ms: PLUGIN_EVENT, TELEPORT_EVENT, POSITION_POLL,
CONSOLE, or TIMEOUT. The external channels (PlayerTeleportEvent,
the pinned position poll) fire on the destination region's tick after
the async chunk load and the cross-region entity handoff, so on Folia
they read the landing 1-2 ticks (50-100 ms) after the plugin issued the
teleport. When the plugin under test is LeafRTP, the harness registers
for its PreTeleportEvent / PostTeleportEvent by class name (no
compile dependency) and completes the attempt from PostTeleportEvent
instead - the plugin's own completion instant. plugin_latency_ms
(Post minus Pre) is the teleport call alone. Competitor arms keep the
external channels and write -1 there. Compare latency_ms across
arms only within one attribution_source.
external_latency_ms (per attempt) is dispatch to the first
PlayerTeleportEvent or position-watch sighting, recorded on every arm
including LeafRTP's. This is the cross-plugin latency column. A LeafRTP row
is held up to 1 s for the sighting and writes -1 if none arrives.
fail_reason=NOT_AT_DESTINATION. LeafRTP fires PostTeleportEvent
whether or not the platform teleport succeeded, so on Post the probe checks
that the player is within 3 blocks (XZ) of the destination, in the same world.
landing_class (with to_world, to_y, landing_floor/feet/head) is
the landing block column read with fixed criteria: SAFE, LAVA, WATER,
SUFFOCATING, NO_FLOOR, HAZARD, VOID, or UNCHECKED_* when it could
not be read without loading a chunk or crossing a region.
main_thread_cpu_scope (phases) is main-thread on Spigot/Paper and
folia-region-threads:N on Folia, where main_thread_cpu_ms sums every
region scheduler thread. Folia rows without this column measured one thread.
cpu_*_ms (phases) split process_cpu_ms by thread name, summed from
per-thread CPU deltas sampled every cpu-breakdown-sample-ms (default
1000) and at phase boundaries:
| Column | Threads |
|---|---|
cpu_server_thread_ms |
the tick thread (Server thread) |
cpu_region_threads_ms |
Folia region scheduler threads |
cpu_scheduler_ms |
Bukkit async workers (Craft Scheduler Thread) |
cpu_scheduler_by_plugin |
the same, as plugin=ms;... from the name Paper gives a worker while it runs a task; (idle) is time between tasks |
cpu_async_scheduler_ms |
Paper/Folia AsyncScheduler workers |
cpu_chunk_system_ms |
chunk load, generation and region I/O workers |
cpu_network_ms |
Netty |
cpu_other_java_ms |
every other Java thread; top 8 names in cpu_other_top |
cpu_non_java_ms |
process CPU minus all of the above: GC, JIT and VM threads, plus the last interval of any thread that exited between samples |
cpu_gc_ms |
the JVM's GC-thread CPU counter, a subset of cpu_non_java_ms; JDK 26+ only, -1 otherwise |
A worker is charged to the plugin named at sample time, so a worker that
switched plugins within one interval bills the whole interval to the later
one. The harness's own async work appears as StressTestRTP. Check
cpu_other_top after a run on a new platform: a large entry there is a
thread family the name patterns in CpuSampler.classify do not cover yet.
chunk-load-cost-us is read per platform family (-paper, -folia, or
the base key on Spigot) with no fallback, so chunk_load_cost_ms /
cpu_ms_with_chunks stay empty on Paper and Folia unless their own key is
set. On those platforms chunk-worker CPU is already in process_cpu_ms
and is measured directly as cpu_chunk_system_ms.
chunks_sync_requested / chunks_sync_by_plugin (phases) name the
plugin that synchronously requested each load, from the ChunkLoadEvent
call stack. They are -1 unless chunks_sync_selftest is PASS (one sync
and one async load of a generated chunk at startup, sync-load-selftest).
chunks_on_tick classifies by firing thread, which Paper and Folia make
near 100% for every plugin; publish chunk counts from
chunks_inclusive_per_attempt.
chunks_landing_area. Paper loads a player's view area after the
teleport event, so those loads used to land on whichever attempt was in
flight, or in background when none was, which made chunks_per_attempt
depend on how long a plugin's teleports take. A load within
viewDistance + 1 chunks (Chebyshev) of an account's last successful
destination is now charged to that teleport until the same account's next
dispatch. The window is never time-based, so one account's consecutive
teleports cannot overlap. These loads are excluded from
chunks_loaded_attributed and chunks_loaded_background, so attributed +
landing area + background = chunks_loaded. chunks_per_teleport is
(attributed + landing area) / attempts.
region_tps_*. Folia has no server-wide TPS; Server#getTPS()
throws, so the tps column there is a wall-clock timer on the global
region, which is never where a teleport lands. It can read 20.0 while
every player region is saturated, and the dips it does show are
global-region hiccups. RegionTpsSampler reads
Server#getRegionTPS(World, cx, cz) for the region each online player
stands in, on region-tps-sample-period-ms, and writes:
| Column | Meaning |
|---|---|
region_tps_5s_at_dispatch (per attempt) |
5 s TPS of the dispatching player's region |
region_tps_scope (per phase) |
FOLIA_PLAYER_REGIONS, SERVER_NATIVE (Paper: tps already covers the server), or GLOBAL_REGION_TIMER (Folia build without the API; tps is global-region only) |
region_tps_samples |
player-region samples in the phase |
region_tps_5s_min / region_tps_5s_mean |
over those samples |
region_tps_1m_min |
lowest 1 m window seen |
region_tps_below_target_fraction |
samples at or under a fixed 19.0 TPS |
These are over player-region samples, not distinct regions - the API
exposes no region id. -1 means NOT MEASURED (off Folia, or before the
first sample). Per-region tick durations are not in the public API; use
the spark integration below for MSPT percentiles on Folia.
Methodology notes¶
- Don't trust a single run. Fire
/rtpstress startat least three times per plugin per server-restart and report median-of-medians. - Pre-generate the world before benchmarking, otherwise you're measuring chunk generation, not RTP.
- Run on identical hardware and identical server JARs when comparing competitors. Differences in tick budget between Paper builds swamp differences between RTP plugins.
- Disable other plugins that hook
PlayerTeleportEvent— they can add latency that the harness will attribute to the plugin under test.
Test consistency protocol¶
The numbers are only comparable if every plugin is measured under the same conditions. Aliasing, JIT warm-up, cache state, and machine noise can each swing a result by more than the difference between two plugins, so pin all of them down before you trust a column. Follow this checklist top-to-bottom for each comparison campaign.
1. Pin the environment (set once, never change mid-campaign)¶
- Hardware: one physical box, no other tenants. Disable CPU frequency scaling / turbo where you can (a thermally throttled box produces a downward MSPT/TPS drift that looks like a regression).
- Server JAR: identical build (
paper-1.20.x-bNNN) and identicalserver.properties(view-distance,simulation-distance,max-tick-time) for every plugin under test. - JVM: identical flags (
-Xms==-Xmxso the heap never resizes mid-run — a heap grow shows up as a false "memory footprint" spike), identical Java version, identical GC. Pin-Xms=-Xmxeven if you normally don't; a resize event aliases straight into the heap series. - Sampler cadence: leave
sample-period-msat the50default (one tick) for every plugin. Changing the cadence between plugins makes the p95-MSPT and peak-heap columns incomparable. - World: pre-generated to a radius larger than the RTP max range, and byte-for-byte identical between plugins (see section 3).
2. Pin the workload¶
- Use
/rtpstress sequence <perTarget> <gap> <concurrency>rather than separatestartruns: it exercises everytarget-commandsentry back-to-back with identical duration, concurrency, and roster, then idlesgapseconds so the box (and any pre-warmed queue) recovers before the next plugin. One sequence run == one apples-to-apples comparison. - Keep
concurrencyandperTargetidentical across campaigns. A longer phase warms more JIT and fills more cache, so a 60 s phase and a 600 s phase are not comparable even for the same plugin. - Keep the roster size constant. RTP latency is per-player; driving 4 alts vs. 1 operator changes the contention profile.
3. Reset state between plugins (the consistency killer)¶
Cache and queue state leaks across runs and is the single biggest source of irreproducible numbers:
- Cold-start column: only meaningful on a freshly restarted server
with an empty queue. Restart the server between plugins (or at least
before the run you read cold-start from) — there is intentionally no
/rtpstress reload, because re-running from a known-fresh state is the only way to get a comparable cold-start. - World reset: if a plugin mutates the world (sets spawn, places blocks), restore the pre-generated world directory from a backup before the next plugin so chunk-load cost is identical.
- Cache reset: configure
cache-reset-commandsfor the plugin's own queue/cache-clear command and run/rtpstress reset-coldbetween phases when you cannot afford a full restart. A restart is always the gold standard;reset-coldis the fast approximation.
4. Warm up, then measure¶
- The first successful attempt after a restart is your cold-start number — record it, then discard the rest of that warm-up phase.
- HotSpot needs a few hundred attempts to inline the hot path. Treat the first warm-up phase as throwaway; read warm-queue / p95-MSPT / peak-heap only from the measured phases that follow.
5. Repeat and aggregate¶
- Run the whole
sequenceat least three times, ideally across three independent server restarts. - Report median-of-medians for latency and worst-case (min TPS, p95 MSPT, peak heap) — averaging away a tick spike hides exactly the pathology the table is meant to expose.
plot_stress.pyre-derives attempt-weighted means across multiple runs; point it at all the run CSVs from a campaign rather than eyeballing a singlesummary.txt.
6. Sanity-check before trusting a column¶
- Min TPS == 20.00 and p95 MSPT well under 50 ms across the whole
run usually means the workload was too light to differentiate
plugins — raise
concurrencyuntil the box is actually under load. - A flat heap series (
mb_per_attempt ≈ 0) means either the run was too short to observe growth or the heap was reset mid-run (an-Xms != -Xmxresize, or a forced GC). Re-check section 1. fail_reason=TIMEOUTon a large fraction of attempts means the target command was rejected without firingPlayerTeleportEvent(wrong syntax, cooldown, permission). Fixtarget-commands/ConsoleWatcherpatterns before reading any latency column — timed-out attempts are not in the latency percentiles and silently bias the comparison.
Spark profiler integration (optional)¶
When the spark profiler plugin is installed,
StressTestRTP automatically brackets each measurement phase with
spark profiler start --timeout N [--only-ticks-over T] [--thread S] /
spark profiler stop --comment <target_label>. One profile is produced
per sequence target (and one per start / burst run), and the spark
upload's comment matches the CSV's target_label column so the two
artifacts correlate 1:1.
This gives you white-box "where did the time go" data (sync chunk loads,
GC pauses, region scans) alongside the harness's black-box per-attempt
timings. Both a spark plugin jar and the spark that Paper 1.21+ bundles
are detected; the bundled one registers no Bukkit plugin, so the hook
also checks for the /spark command. When neither is present the hook
logs one line and no-ops — no hard dependency.
Configure under spark: in config.yml (defaults: enabled, 90 s
timeout, only-ticks-over 50 ms). Set spark.enabled: false if you'd
rather drive /spark profiler manually.
spark.threads(defaultServer thread,RTP-Anvil-IO-*) is a comma list; each entry becomes its own--thread <name>(names with spaces are not quoted; spark re-joins them). An entry containing*is a wildcard and adds--regex(exact entries are escaped), so pool threads spawned after the profile starts are still sampled.*alone samples every thread;""omits the flag (server thread only).- With
save-to-file, spark writesplugins/spark/profile-<stamp>.sparkprofile(the log line names the folder that was found). The auto-summary polls that folder every ~1 s, starting afterauto-summary-delay-ticks, untilspark.auto-summary-timeout-seconds(default 30) have passed since the stop. It accepts only a profile that is new since the stop and whose size is unchanged across two polls. Spark's save can take several seconds on Windows.
Future work (out of scope for v1)¶
- Out-of-process bot driver (a sibling helper, e.g.
helpers/StressTestRTPBots/) — connects N protocol-level bots to the test server so the roster scales past the alts the operator has. - Automated competitor sweep — a wrapper script that swaps
target-command, restarts the server, and concatenates summaries.
Both follow once v1 has produced a baseline run for RTP and RTP-Pro.
Non-goals¶
- No GUI. Plain commands and plain CSV/TXT files.
- No live charts / web dashboard. The CSV is the artifact; analysis belongs in a notebook.
- No bypass of permissions / cooldowns / economy. Those are part of what's being measured.
- No interaction with
rtp-apiorrtp-core. This is a black-box harness by design — it must work identically against RTP and against every competitor.
If you want any of the above, fork this plugin or open a discussion before bolting flags onto it.