rtp-proxy-ADR-015 - Shared Network Waitlist and Dynamic Batched Dispatch¶
Status: Accepted (2026-05-21; Slice 1 Slice 2 landed 2026-05-21)
Refines: rtp-proxy-ADR-014, rtp-proxy-ADR-001
Depends on: leafrtp-ADR-022, REQ-RTP-NET-015
Tracking: docs/dev/scratch/CHECKLIST-network-waitlist.md
Context¶
ADR-014 (L6) defined the cross-server /rtp dispatch path:
lobby /rtp -> NetworkRouter -> NetworkRequestQueue (Redis BLPOP)
-> proxy DefaultRtpDispatcher -> ReservationClient.claim
-> transferPlayer -> backend redeemReserved
Today, when DefaultRtpDispatcher cannot place the player on a backend right now (every peer is kill-switched, the requested region has no peer with networkKeptCount > 0, the rate limit is exhausted, or BackendSelector returns no qualifying backend), the request resolves as FAILED and the lobby surfaces a configurable terminal message. The player must re-invoke /rtp until a backend is free.
This contradicts the per-backend /rtp UX, where RegionQueueManager.playerQueue notifies the player ("you are #N in queue"), locks further /rtp commands for that UUID until served, and drains in a bounded batch per QueueTask pulse. The local UX is the contract operators expect; the network UX should mirror it.
Two cross-cutting constraints distinguish the network case from the single-backend case:
- The waitlist is shared across proxies. Multiple proxies (REQ-RTP-NET-014) read the same coordination store; a player who enrols through
proxy-1must be removable byproxy-2ifproxy-2is the one that learns about the disconnect. - Available capacity is dynamic and per-backend. Each backend publishes
BackendHeartbeat.networkKeptCountonce per heartbeat interval; a drain pulse should take exactly as many envelopes from the head of the waitlist as the qualifying backends can absorb right now (sum of their kept counts), no more, no less. A fixed batch-size knob would either starve the waitlist or oversubscribe a backend whose reserve was just drained.
Decision¶
Introduce a new sibling SPI to NetworkRequestQueue, named NetworkWaitlist, alongside two implementations:
InMemoryNetworkWaitlist- reference impl, single-JVM, devstack / unit tests /transport.kind = memoryfallback.RedisNetworkWaitlist- production impl, atomic via Lua scripts. Landed in Slice 3b (2026-05-21): seven Lua scripts (waitlist_enrol,waitlist_peek,waitlist_drain_batch,waitlist_position,waitlist_remove_uuid,waitlist_reap,waitlist_refresh_ttl) undersrc/main/resources/redis/, each with a SHA1 sidecar verified at load (LF-normalized). Keyspace:rtp:net:waitlist:list(FIFO LIST), rtp:net:waitlist:uuids(SET for O(1) duplicate guard and point-remove),rtp:net:waitlist:cids(SET for correlationId idempotency). Drain is two-step under theWaitlistLeaderLeasesingle-drainer guarantee:waitlist_peek(non-mutating LRANGE) followed by client-side per-backend allocation followed bywaitlist_drain_batch(atomic LREM per allocated correlationId). Verified by no-RedisRedisNetworkWaitlistScriptLoadTest(sidecar parity) and opt-inRedisNetworkWaitlistIT(12 cases,RTP_REDIS_IT=truegated).
The dispatcher rewiring (DefaultRtpDispatcher parks unservable envelopes), the proxy-side drainer with leader-lease election, and the lobby-side notify + command-lock + quit-removal wiring are Slices 2-4 of the checklist and are described here at interface level only - their concrete implementation slices will not change the SPI shape this ADR introduces.
SPI Shape (Slice 1, this ADR)¶
public interface NetworkWaitlist {
enum EnrolOutcome { ACCEPTED, REJECTED_FULL, REJECTED_DUPLICATE, REJECTED_CLOSED }
enum CancelReason { PLAYER_DISCONNECT, TTL_EXPIRED, EXPLICIT_REQUEST,
BACKEND_UNAVAILABLE, TRANSPORT_ERROR }
record WaitEnvelope(UUID playerId, UUID correlationId,
Optional<String> regionKey, Optional<String> serverHint,
String originServerId, long enrolledAtMs) { }
CompletableFuture<EnrolOutcome> enrol(WaitEnvelope env);
CompletableFuture<Map<String, List<WaitEnvelope>>> drainBatch(Map<String,Integer> perBackendCaps, int globalCap);
CompletableFuture<Optional<Integer>> position(UUID playerId);
CompletableFuture<Boolean> remove(UUID playerId, CancelReason reason);
CompletableFuture<Integer> reap(Duration maxAge);
CompletableFuture<Integer> size();
}
Five design points:
- Sibling, not subtype.
NetworkRequestQueueis BLPOP-shaped (one dequeue per call, no random access). The waitlist needs position-by-UUID, point-remove-by-UUID, and batch-drain-with-per-backend-caps - none of whichNetworkRequestQueueexposes. A new SPI is cheaper than overloading the existing one and leaves ADR-014's hot path unmodified. - At-most-one entry per
playerId.enrolrejects withREJECTED_DUPLICATEif the UUID already has a live entry under a differentcorrelationId. This is the contract the lobby-side command lock depends on - "is the player on the waitlist?" reduces to "doesposition(uuid)resolve non-empty?". - Idempotency by
correlationId. A replay of the same envelope resolvesACCEPTEDwithout a second insert. The lobby may retry anenrolcall after a transient transport error without risking a duplicate-rejection on the second try. drainBatchsized by the caller. The SPI does not know whatnetworkKeptCountis, what a heartbeat is, or what aBackendSelectoris. The drainer passes aMap<serverId, cap>snapshot computed from the latest heartbeats; the SPI's job is FIFO-preserving allocation across that map, head-blocking when no backend qualifies for the head envelope's pinned region. TheglobalCapargument is a defence-in-depth ceiling (default = sum of per-backend caps).removeis point-keyed byplayerId, notcorrelationId. The lobby'sPlayerQuitEventknows the UUID; it does not necessarily know which correlationId was active. Keying remove by UUID matches the disconnect-correction signal the operator gave during proposal.
In-Memory Reference Impl¶
InMemoryNetworkWaitlist backs the SPI with a LinkedHashMap<UUID, WaitEnvelope> for O(1) duplicate-rejection, O(1) point-remove, and FIFO position / drainBatch traversal. All access is serialized through a single-thread executor (the same pattern InMemoryNetworkRequestQueue uses), so every public method is atomic from the caller's perspective. maxSize is a constructor knob (default 1024); over-capacity enrolments resolve REJECTED_FULL rather than throwing or silently dropping - this preserves S-004 attribution at the SPI boundary.
The reference impl's drainBatch selection is deterministic (smallest-keyed serverId with cap > 0 wins each round). Production impls are free to choose differently (round-robin, weighted, region-pinned). Determinism in the reference impl is a test-stability decision, not a public contract.
Status Coverage (Slice 2, landed)¶
ADR-014's NetworkRequestQueue.QueueState was extended with a new WAITLISTED value (rather than reusing QUEUED as the Slice 1 ADR draft suggested) so the lobby-side NetworkStatusCache (Slice 4) can distinguish "parked on the shared cross-proxy waitlist, awaiting capacity" from "claimed and dispatched into the per-backend queue". QUEUED retains its ADR-014 meaning ("envelope on the per-backend FIFO awaiting dequeueReady"). The new value is documented in NetworkRequestQueue.QueueState and emitted by DefaultRtpDispatcher via StatusSink whenever the no-backend branch parks onto a configured NetworkWaitlist.
Total-Failure Backoff (Slice 2, landed)¶
When NetworkWaitlistDrainer drains a batch and every drained envelope fails dispatch (no backend qualified, all kill-switched mid-flight, transport partition), the network is unable to serve right now. The drainer responds with:
NetworkWaitlist.refreshAllTtl()- refreshes every live entry'senrolledAtMsto the current wall clock so the subsequentreap(maxAge)sweep does not silently age entries out due to time spent waiting on a dead network.- A pause window (
pauseFor, default 2 s) during which subsequentrunPulse()calls short-circuit toSKIPPED_PAUSEDrather than hammer a known-bad network.
This is the "if it fails on all servers in order of priority, refresh all TTL in the queue and pause" semantic the issue requested.
Leader Election (Slice 2, landed; Redis impl Slice 3a, landed)¶
WaitlistLeaderLease is a sibling SPI to NetworkWaitlist. The reference impl AlwaysLeaderLease is sufficient for single-proxy and in-memory devstack deployments; RedisLeaderLease (Slice 3a, landed 2026-05-21) uses SET key holderId NX PX <ms> for first acquire, SET XX PX for same-holder re-extend (idempotent per SPI contract), and a compare-and-DEL Lua for release so a stale holder cannot stomp a successor's lease after TTL handover. Verified by opt-in RedisLeaderLeaseIT (8 cases, RTP_REDIS_IT=true gated).
Lobby-side UX (Slice 4, landed 2026-05-21)¶
The lobby (rtp-plugin) consumes the proxy-emitted WAITLISTED state via three new pieces wired into NetworkModeBootstrap:
NetworkStatusCache.QueueStatus.State.WAITLISTEDwas added alongside the existingQUEUED/ROUTING/RESERVED/TRANSFERRING/terminal values. The bootstrap-side adapter atNetworkModeBootstrapline ~360 was alreadyState.valueOf(proxyStatus.state().name()), so proxyWAITLISTEDrows now flow through without an explicit branch. A newQueueStatus.nonTerminal()helper centralizes the truth table used by the guard.NetworkWaitlistGuard implements Predicate<CommandSender>is registered onRTPCmdBukkitviaaddSenderCheck(...). While the caller's cache entry is non-terminal (any ofQUEUED,WAITLISTED,ROUTING,RESERVED,TRANSFERRING),/rtp*invocations short-circuit with a configurable message resolved fromMessagesKeys.alreadyQueued(placeholder[position]). See "Locale and Message Coverage (Slice 5)" below.NetworkWaitlistNotifierruns onRTP.scheduler.runTaskTimerAsynchronouslyat the same cadence as the existing status-cache poll. Per pulse it walks the cache snapshot, emitsmsgNetworkQueued-style messages forWAITLISTEDrows, and dedupes per-UUID by(body, lastEmittedAtMs)so the player only sees a new line when their position changes or the configurednetwork.waitlist.notifyIntervalSeconds(default 5) elapses. Dedup entries for players evicted from the cache or transitioned out ofWAITLISTEDare dropped on the next pulse.NetworkWaitlistQuitListener implements Listeneris registered alongside the existingJoinTriggerSource(siblingregisterWaitlistQuitListener(plugin)method). OnPlayerQuitEventit callsNetworkRequestQueue.cancel(uuid, PLAYER_DISCONNECT); that SPI op is already idempotent and safe when there is no live entry. Failures are logged viaRTP.logper S-004 but never propagate out of the event dispatch.
The whole UX surface is gated on network.enabled = true and on boot() reaching status-cache wiring; on single-server installs (or when the bootstrap's no-op path is taken) the guard is null, the notifier is null, the listener is null, and the existing /rtp path is byte-for-byte unchanged.
Verified by ReqRtpNet015NetworkWaitlistTest (Slice 4 REQ-traceable suite: state truth table, guard rendering, notifier dedup / interval / position-change / cache-eviction / state-transition, quit-listener constructor contract, and a pinned name()-mapping sanity test guarding against silent enum drift between the proxy and lobby state enums).
Locale and Message Coverage (Slice 5, landed 2026-05-21)¶
The user-facing message keys land as three new MessagesKeys enum entries (alreadyQueued, networkTimedOut, waitlistFull); the 4th planned key networkQueued already existed from a prior pass. Baseline messages.yml rows added with English values and leading comment blocks. NetworkWaitlistGuard.formatMessage(...) and NetworkWaitlistNotifier.renderBody(...) now resolve their templates from RTP.configs.getParser(MessagesKeys.class).getConfigValue(...), with hardcoded English defaults retained only as a fallback for early-bootstrap / test contexts. The previous dual constants for "position known" vs "position unknown" collapsed to a single [position]-substitution path (empty substitution when the proxy has not yet assigned one). Locale TSV pipeline (scripts/locale-files-to-csv.ps1 -> scripts/reconcile-locale-csvs.ps1 -> scripts/locale-files-from-csv.ps1) was run end-to-end so every shipped locale carries the three new keys identity-seeded from English per TRANSLATION_GUIDE.md section 8; native-speaker review remains a routine locale follow-up.
Consequences¶
Positive.
- The cross-server UX matches the per-backend UX one feature at a time (notify, lock, batch). Operators do not have to learn two divergent queue models.
- The waitlist is additive: when
network.mode = offortransport.kind = memoryand no drainer is started, the SPI is unreferenced and the ADR-014 hot path is byte-for-byte unchanged. BackendHeartbeat.networkKeptCount(added in ADR-014) becomes the load signal for batch sizing without any new wire-format addition; the heartbeat is already polled byBackendSelector.- Per-player point-remove on
PlayerQuitEventremoves a known operational sharp edge: under ADR-014 alone, a player who quits before their reservation lands leaks aReservationTokenuntil the proxy reaper expires it. The waitlist's point-remove path closes that window for the parked-but-not-yet-dispatched case.
Negative.
- One more SPI surface (
NetworkWaitlist) and one more proxy daemon (NetworkWaitlistDrainer, Slice 2) for operators to learn. Mitigated by the in-memory impl being the devstack default and the production Redis impl deferring to a later slice. - Two cooperating data structures (
NetworkRequestQueueandNetworkWaitlist) instead of one. The dispatcher must choose at request time which to write to; the choice istry-claim, on-no-backend write-waitlist, but the policy is now policy-in-code rather than policy-in-config. Justified by the SPI mismatch above. - The Redis impl (landed Slice 3b, 2026-05-21) introduces seven Lua scripts and the
rtp:net:waitlist:*keyspace. Script identity is pinned by checked-in.sha1sidecars per the project's existing convention; a whitespace edit that fails to regenerate the sidecar refuses to enable at startup.
Risk-mitigated.
- A single-backend network sees no behavior change: the lobby still dispatches every request via the existing FIFO, and the waitlist remains empty.
- The
maxSizecap prevents an unbounded-growth pathology when every backend is down for an extended outage; over-capacity enrolments fail loudly (REJECTED_FULL) rather than silently consuming proxy memory.
Alternatives Considered¶
- Extend
NetworkRequestQueuewith aparkUnreadyoperation. Rejected: would force every existing impl (InMemoryNetworkRequestQueue,RedisNetworkRequestQueue,SqlNetworkRequestQueue) to grow position-query and point-remove semantics they do not need for the FIFO path, doubling each impl's surface area. - Per-backend wait queues (one queue per backend serverId). Rejected: a player who requested region
miningand is parked behind backend-A's queue cannot benefit when backend-B suddenly frees aminingslot. A single FIFO with caller-supplied per-backend caps preserves cross-backend fairness. - In-process per-lobby waitlist (no shared store). Rejected by the operator during proposal: lobbies are not the authority on which proxy will eventually serve the player, and a lobby restart would forget the waitlist. The shared-store model is the explicit ask.
References¶
docs/dev/scratch/CHECKLIST-network-waitlist.md- slice plan.- REQ-RTP-NET-015 - shared network waitlist requirement.
- rtp-proxy-ADR-014 - cross-server dispatch baseline (refined here).
rtp-proxy-common/src/main/java/.../spi/NetworkWaitlist.java- SPI source.rtp-proxy-common/src/main/java/.../transport/memory/InMemoryNetworkWaitlist.java- reference impl.rtp-proxy-common/src/test/java/.../transport/memory/InMemoryNetworkWaitlistTest.java- SPI conformance tests.