rtp-proxy-ADR-014 - Backend-Owned /rtp with Network Wait-Queue (L6)¶
Status: Accepted (2026-05-21)
Refines: rtp-proxy-ADR-001, rtp-proxy-ADR-005, rtp-proxy-ADR-011
Supersedes (in part): rtp-proxy-ADR-006 section Command Hosting - the Velocity adapter no longer registers a Brigadier /rtp; the proxy's role is reduced to dispatch and reservation lifecycle.
Depends on: leafrtp-ADR-022, MULTI_SERVER_PLAN.md L6 row.
Context¶
The L1-L5 phases of the multi-server work assumed the proxy hosts /rtp directly: a CommandTriggerSource in rtp-proxy-common parsed the slash-command, the proxy chose a backend via BackendSelector, claimed a ReservationToken, and pushed the player through ServerPreConnectEvent. This shape produced four operational problems that L6 is written to fix:
- Backends own the cache, the proxy owns the request. All pre-warmed coordinates live in
RegionQueueManageron the backend (keptLocations,unkeptLocations,backlogLocations). The proxy can route a player to a backend, but it has no way to earmark "this specific coordinate from that backend's pool is mine for the next 30 seconds". The result is a race: the proxy claims, the backend keeps serving its local/rtpqueue, the player arrives, and by the timeJoinTriggerSourcefires the warmed coord is already gone. - Two writers to the same store. Both the proxy (claim/release) and the local backend
/rtp(poll) mutatekeptLocations. There is no protocol for "this slot is yours for the cross-server case, hands off". A backend cannot tell, from its own perspective, which of its kept coords are spoken for. - Per-region partitioning has no proxy expression. Region selection (
/rtp [region]) is a backend concept enforced byRegionVerbosePermissionand the per-region cache; on a multi-backend network there is no shared schema for "backend A serves regiondefault, backend B serves regionnether, both servemining". The selector'sregionsAvailable: List<String>is best-effort, with no per-region kept-count visibility. /rtpon the proxy is a duplicate command surface. Operators and addons already integrate against the backend's/rtp(cooldown, economy, claim-plugin hooks,commands-apiBrigadier bridge, locale messages). Reproducing this surface on the proxy duplicates code and creates two divergent permission/policy paths.
Decision¶
L6 inverts the model: the backend retains ownership of /rtp, the proxy is reduced to a dispatch and reservation broker, and a new transport-backed FIFO carries cross-server requests from the originating backend through the proxy to the destination backend.
Architecture¶
Player runs /rtp on backend A
-> backend A NetworkRouter decides: local-serve | cross-server | reject
(gates: kill-switch, killSwitchSafetyTtl, requestsPerSecond+burst,
regionAvailability, region collision policy, mode=auto+local-able,
backend-A keptCount > 0)
-> if local: existing local pipeline (unchanged)
-> if cross-server:
backend A enqueues EnrolmentEnvelope into NetworkEnrolmentBuffer
NetworkRequestQueue.flushPending(...) -> proxy-visible FIFO
proxy worker (TransportRequestTriggerSource) BLPOPs envelope
proxy DefaultRtpDispatcher:
- picks backend B via BackendSelector (region + keptCount aware)
- calls ReservationClient.claim on backend B's earmarked coord
- fires StatusSink: RESERVED -> ROUTING
- transferPlayer(playerId, B)
- fires StatusSink: COMPLETED on ServerPreConnectEvent.success
backend B JoinTriggerSource:
- on PlayerJoinEvent, redeemReserved(networkTokenId)
- if non-null, acceptRedeemedReservation -> personal queue gets the coord
- existing /rtp dispatch on backend B picks up that exact coord
- on PlayerQuitEvent: releaseToNetworkKept + transport.release(PLAYER_DISCONNECTED)
ReservationTokenReaper (proxy-side):
- periodically reaps expired tokens
- ReleaseSink fires per-token (in-process for co-located dev; cross-process channel
deferred to a future ADR - backends learn TTL_EXPIRED via their existing claim flow
in steady state)
Data Structures (backend, rtp-core)¶
RegionQueueManager gains two sibling pools alongside keptLocations:
| Field | Type | Lifecycle |
|---|---|---|
networkKeptLocations |
LockFreeLocationBuffer (nullable) |
Cross-server reserve carved from cacheCap; capped at min(networkReserveSize, cacheCap). 0 disables. |
networkReservedLocations |
ConcurrentHashMap<UUID, RTPLocation> |
In-flight reservations keyed by token UUID. |
Six new public APIs on RegionQueueManager:
reserveFromNetworkKept(UUID tokenId): RTPLocation- pops fromnetworkKeptLocations, parks innetworkReservedLocations, idempotent per token (re-reserves return the parked value).redeemReserved(UUID tokenId): RTPLocation- removes fromnetworkReservedLocations, hands coord to caller (consumes the reservation).releaseToNetworkKept(UUID tokenId): boolean- returns the coord tonetworkKeptLocations; on bounded-pool full, falls through tounkeptLocations(S-004 attribution: never silently dropped).acceptRedeemedReservation(UUID playerId, RTPLocation loc): void- re-offers a redeemed coord into the player's personal queue so the immediately-following local/rtpdispatch picks that exact coord (the "teleport directly" semantics from the checklist, implemented via the existing personal-queue mechanism rather than a parallel teleport pipeline).keptCount(): int,networkKeptCount(): int,networkReservedCount(): int- observability forBackendHeartbeatpayloads andBackendSelectordecisions.
Per-Region Split Knob (decided during implementation)¶
The size of networkKeptLocations is set per-region via a new RegionSettings record component networkReserveSize (Long, default 0, clamped to cacheCap). 0 means "feature off for this region" and is the default on existing servers - Slice A ships with no user-visible behavior change until an operator explicitly opts in via regions.yml. The alternative networkReservePercent (ratio) was considered and rejected: absolute slot counts mirror the existing cacheCap / activeChunkCap / backlogCacheCap style, are simpler to validate (<= cacheCap), and are immune to surprising auto-rescale when cacheCap changes. A networkReserveSizeMode = ABSOLUTE | PERCENT_OF_CACHE_CAP knob can be layered on later without breaking the field name.
Proxy SPI Additions (rtp-proxy-common)¶
NetworkRequestQueue- new SPI with six methods (enrol,flushPending,pollStatus,dequeueReady,transition,cancel), three impls (InMemoryNetworkRequestQueue,RedisNetworkRequestQueue,SqlNetworkRequestQueue). All async via single-thread daemon executors; correlationId-keyed idempotency; terminal-state cleanup of envelope rows.- Four Redis Lua scripts under
src/main/resources/redis/with.sha1sidecars:enqueue_batch.lua(atomic SADD+RPUSH+HSET),pollStatus.lua(HGETALL + LPOS-derivedpositionInQueue),dequeueReady.lua(LPOP + envelope HGETALL + status -> ROUTING),transition.lua(state UPSERT; on terminal: DEL env HASH + SREM seen). StatusSink-@FunctionalInterfacehook onDefaultRtpDispatcherfiring at every terminal outcome (RESERVED / COMPLETED / FAILED / CANCELLED). The originating backend'sNetworkStatusCachereads viaNetworkRequestQueue.pollStatus; this sink is the proxy-side write half.ReleaseSink-@FunctionalInterfacehook onReservationTokenReaperfiring per reaped token (in-process; backend co-located with reaper in dev, cross-process channel deferred to a future ADR - the steady-state path is covered byJoinTriggerSource-on-quit callingtransport.release(..., PLAYER_DISCONNECTED)directly).BackendHeartbeatextended with four L6 fields:keptCount,networkReservedCount,regions: Set<String>,regionKeptCounts: Map<String,Integer>. Legacy 13-arg / 14-arg ctors retained for back-compat.BackendSelectorextended withchoose(req, snap, serverIdFilter)for exact-match pinning.WeightedAverageBackendSelector.qualifies(...)now excludes kill-switched peers; region-aware filtering prefersregions: SetwithregionsAvailable: Listfallback, and gates onregionKeptCounts.getOrDefault(key, 0) > 0only when the map is populated.NetworkTransport.listActiveForServer(serverId)- new default-empty SPI method returningCompletableFuture<List<ReservationToken>>. Used at backend boot to repopulatenetworkReservedLocationsfrom the shared store after a crash, releasing stale tokens viaBACKEND_REJECTED. One-shot reconcile; no steady-state pulse (the steady-state hot path is fully covered byredeemReservedon join +releaseToNetworkKepton quit).
Backend Wiring (rtp-plugin)¶
A new io.github.dailystruggle.rtp.bukkit.network package hosts:
NetworkRouter- 7-gate decision matrix (kill-switch, kill-switch safety TTL, rate limit, region availability, region collision, mode=auto+local-able, backend keptCount > 0). Lazy-refillTokenBucketfor the rate-limit gate.Mode.parseand region-arg parsing reject=/:per the cross-server arg-shape contract.NetworkEnrolmentBuffer-ConcurrentLinkedDequedirty-write buffer; on sink failure, head-re-enqueues the failed batch per S-004 (no silently dropped enrolments).NetworkStatusCache- 8-state wholesale-replace cache; on supplier failure, the last-known state is preserved per S-004.NetworkRegionCollisionWarner- boot-timeC-warnfor ambiguous region names across backends; forwards-compatiblePolicy.parse.NetworkModeBootstrap- wires the four components above, opens theNetworkRequestQueueviaNetworkBindings.openRequestQueue, runsreconcileNetworkReservations(transport, myServerId)once at boot, and adds forward-order teardown. Failure-mode policy: on wiring exceptions the bootstrap fails open (router/buffer/cache nulled, transport stays up, local/rtpstays functional).JoinTriggerSource-REDEEMEDoutcome from the proxy'sServerPreConnectEventredeem hook now callsRegionQueueManager.redeemReserved(...)acrosspermRegionLookup+tempRegions, thenacceptRedeemedReservation(playerId, coord)to pin the redeemed coord onto the player's personal queue, with S-004 release-on-accept-failure.PlayerQuitEventhandler releases the bound coord and CAS-transitions the proxy-side token toPLAYER_DISCONNECTED.
Velocity Adapter (rtp-proxy-velocity)¶
The Brigadier /rtp command is deleted (registerRtpCommand, CommandTriggerSource, onCommandTrigger, rtpCommandMeta). It is replaced by a TransportRequestTriggerSource in rtp-proxy-common running N (network.queue.workerThreads, default 1) daemon BLPOP workers that hand each QueueEnvelope to DefaultRtpDispatcher.dispatch. Shutdown is bounded to 2000ms (interrupt + join + log-on-timeout). The ServerPreConnectEvent reservation-redeem path is preserved.
Consequences¶
Positive:
- The hot path (already-online player runs
/rtpon their current backend) is unchanged - no proxy round-trip for the common case, no new failure modes for single-backend deployments. - Cross-server requests respect per-region kept-count visibility; the selector can refuse a peer that has the region but a starved cache.
- Backend retains the existing
/rtppolicy surface (cooldown, economy, claim-plugin hooks,commands-apiBrigadier bridge, locale messages) - operators see one/rtppermission tree, not two. - The proxy command surface shrinks: one fewer Brigadier registration, one fewer permission key (
rtp.cross), one fewer locale message file on the proxy side. S-004discipline is preserved through every new code path: enrolment-buffer head re-enqueue, status-cache last-known preservation, release-fall-through tounkeptLocations, sink throws caught at the boundary.
Negative:
- One more SPI surface (
NetworkRequestQueue) and one more keyspace (rtp:net:wq:*) for operators to learn. Mitigated by the in-memory reference impl being the default (transport.kind = memoryworks out of the box for dev/test). - The proxy is no longer the single source of truth for "who has been served"; that lives across the proxy's
rtp:net:wq:status:*HASHes and each backend'snetworkReservedLocations. Reconciliation on backend boot (listActiveForServer) handles crash-recovery; steady-state divergence is bounded by the reservation TTL. BackendHeartbeatpayload grows by four fields. Wire-format compatibility maintained via legacy ctors.
Risk-mitigated:
- A backend that opts in to
networkReserveSize > 0but is the only backend on the network sees no change in observable behavior - local/rtpcontinues to pollkeptLocations(the unsplit pool), andnetworkKeptLocationsslots simply sit idle. The split is feature-off-by-default.
Alternatives Considered¶
- Keep
/rtpon the proxy, add a "snapshot the backend's cache through the heartbeat" channel. Rejected: the heartbeat is a steady-state poll, not a transactional store; coordinates would be stale by the time the proxy claimed against them, recreating the original race. - Push-based reservation channel (proxy -> backend on every claim). Considered for
ReleaseSink. The cross-process push needs either a new per-backend inbound list (rtp:net:claimed:<serverId>) and a backend BLPOP worker, or a transport-specific notification primitive (Redis pubsub, PostgresLISTEN/NOTIFY). Deferred to a future ADR; the in-processReleaseSinkis the correct interface, and the steady-state path is covered today byJoinTriggerSource.onQuitcallingtransport.release(...)directly. - Steady-state
NetworkReservationPulseperiodic poll. Considered for backend-side reconciliation. Replaced by one-shot boot reconcile (listActiveForServer-> reserve-or-release): the steady-state hot path is fully covered byredeemReservedon join +releaseToNetworkKepton quit; the pulse only earns its keep on backend restart with outstanding reservations.
References¶
docs/dev/MULTI_SERVER_PLAN.md- L6 row.- leafrtp-ADR-022 - umbrella network-mode decision.
- rtp-proxy-ADR-006 - Velocity bootstrap (partially superseded: section Command Hosting).
- rtp-proxy-ADR-005 - Redis binding keyspace conventions reused for
rtp:net:wq:*. - rtp-proxy-ADR-011 - SQL pool sharing for
SqlNetworkRequestQueue. - REQ-RTP-NET-008 - network wait queue.
- REQ-RTP-NET-011, REQ-RTP-NET-012, REQ-RTP-NET-014 - reservation tokens.