rtp-proxy-ADR-004 — Weighted-Average BackendSelector¶
Status: Accepted
Accepted: 2026-05-14
Date: 2026-05-13
Amended: 2026-05-22 (curve catalogue + configurable scoring table landed in rtp-proxy-common.selector)
Amended: 2026-05-22 (region-pair scoring; lobby and proxy unified on RegionAwareSelector)
Refines: leafrtp-ADR-022
Depends on: rtp-proxy-ADR-001, rtp-proxy-ADR-002
Context¶
ADR-036 locks the single configurable strategy principle: no discrete strategy: leastLoad | roundRobin | random enum. Instead, every candidate backend is scored by a weighted sum of normalised, curved metric values; lowest score wins. This ADR pins the reference selector that lives in rtp-proxy-common.selector and implements REQ-RTP-PROXY-COMMON-002…004 and REQ-RTP-NET-010.
Decision¶
Scoring Formula¶
For each backend b not filtered out:
score(b) = ( Σ_i weight_i * curve_i( normalize_i( metric_i(b) ) ) ) / backendWeight(b)
weight_ifromloadBalancer.weights.<metric>(ADR-002). Zero weights skip the term entirely.normalize_imaps the raw metric to[0.0, 1.0]against a per-metric reference scale (e.g.,mspt / 50.0,queueDepth / max(1, softCap),heapUsed / heapMax).curve_iis one of{linear, exponential, logarithmic, sigmoid, step, power}with parameter boundsk ∈ [0.1, 20],p ∈ [0.1, 8],threshold ∈ [0.0, 1.0].backendWeight(b) = max(0.01, loadBalancer.backends.<serverId>.weight ?? 1.0)— higher means preferred (divides final score).
Lowest score wins. Ties broken by serverIdAsc.
Candidate Filtering (pre-scoring)¶
A backend is excluded before scoring if any of:
lastSeenEpochMsolder thansnapshot.timestamp - loadBalancer.staleAfterMs(REQ-RTP-PROXY-COMMON-003).pluginState != READYoracceptingRequests == false(availability axis).request.regionKeynot inregionsAvailable[](when specified).request.worldKeynot inworldsLoaded[](when specified).schemaVersionoutside the supported range (REQ-RTP-NET-009).- Backend is in the per-proxy cooldown map (recent rejection, until
cooldownMselapsed).
If filtering leaves zero candidates, choose returns Optional.empty() and the dispatcher decides queue-vs-fail.
Hot-Spot Avoidance: recentPicks¶
Per-proxy decaying counter (halflife = 10s default) added to each backend's score:
score(b) += recentPicksWeight * decay(recentPicks[b], halflife)
Local-only by default (REQ-RTP-PROXY-COMMON-008). A recentPicks.mode: shared config knob is reserved for v2 but unimplemented in v1 (would add one transport round-trip per pick).
Fallback Chain (Capped-Retry)¶
Owned by ReservationClient (ADR-001) but parameterised here:
maxRetries: 3— total attempts including the first.attemptTimeoutMs: 1500— per-attempt deadline; on timeout, backend enterscooldownMsand the next-best candidate is tried.cooldownMs: 2000— hysteresis after rejection/timeout/claim-race loss.- Score-sticking — within one in-flight request, the snapshot is read once; subsequent retries score against the same snapshot to avoid mid-flight oscillation (Linux CFS-inspired).
- On exhaustion:
Failed(reason=NO_BACKEND, messageKey="rtp.network.failed").
Determinism & Testability¶
chooseis a pure function of(RtpRequest, NetworkSnapshot, LoadBalancerConfig, recentPicksSnapshot). The selector exposes ascore(b, snapshot)debug accessor for golden-file tests.- Curve plots are rendered for the admin docs via
scripts/lb-curve-plot.py; output lands underdocs/admin/proxies/LOAD_BALANCING.md(Phase 3).
playerCount in v1¶
weight: 0.0 by default. The metric is published in telemetry so post-Phase-2 analysis can re-evaluate, but it does not influence selection in v1. Justification: player count is a lagging indicator vs. mspt/queueDepth and can mask hot spots when several players join the same backend simultaneously.
Alternatives Considered¶
| Alternative | Why Rejected |
|---|---|
Discrete strategy enum (leastLoad, roundRobin, random) |
Forces a config-time choice between objectives that should compose; operators want hybrid behaviour. |
| Lexicographic ordering (sort by mspt, then queueDepth, then …) | Brittle near ties; can flap on noise; no smooth fallback. |
| Reinforcement-learning selector | Out of scope; non-deterministic; impossible to reason about in incidents. |
| Power-of-two-choices random sampling | Simple but ignores regionsAvailable[] and worldsLoaded[]; cannot satisfy REQ-RTP-NET-005 cleanly. |
Consequences¶
- Positive: one selector to test, document, and tune. Operators get full curve control without code changes.
- Negative: more knobs to understand. Mitigation: the bundled defaults (ADR-002) are tuned for the headline goal; the admin docs ship rendered curve plots in Phase 3.
Implementation (2026-05-22)¶
The 2026-05-22 amendment landed the full curve catalogue and the configurable scoring table:
selector/curve/CurveSPI +Curvescatalogue:linear,exponential(k),logarithmic(k),sigmoid(k, x0),step(threshold),power(p). Parameter bounds (k in [0.1, 20],p in [0.1, 8],x0/threshold in [0, 1]) are enforced via clamping; out-of-range inputs are clamped to[0, 1].selector/MetricInputenum (one normalized extractor per input). Supersedes the v1 hardcoded four-term sum:- Original ADR set:
mspt(normalized by 50 ms),queueDepth(bymax(1, softCap)),heapUsed(byheapMax),playerCount(by 100). - Added on 2026-05-22:
heapFree(alias for the used-ratio expressed as "remaining capacity"),keptCount(inverted, normalized by 64),tps(derived frommsptas1 - min(20, 1000/mspt) / 20;BackendHeartbeatdoes not yet publish TPS directly). selector/ScoringTermrecord(MetricInput input, double weight, Curve curve)is the row of the operator-facing scoring table.LoadBalancerConfigcarries aList<ScoringTerm> termsfield. When empty (e.g. when constructed via the pre-curve six-arg legacy constructor), the compact constructor synthesizes alinearterm per non-zero legacy weight, preserving binary and behavioural compatibility for every existing call site (LoadBalancerConfig.defaults()and the existing test suite).LoadBalancerConfigYaml.fromMap(Map<String, Object>)parses both the newloadBalancer.termssub-configuration block (nestedcurve:sub-map, no inline-brace YAML) and the legacy flat-weight keys. Unknowninputorcurve.typevalues raiseIllegalArgumentExceptionwith a descriptive message instead of silently degrading.- Operator documentation: commented-out reference block in
rtp-proxy-velocity/src/main/resources/network-proxy.yml. The Velocity bootstrap still passesLoadBalancerConfig.defaults()at the time of this amendment; wiring the YAML loader throughRtpVelocityPluginis a follow-up tracked outside this ADR.
Coverage: CurvesTest, WeightedAverageBackendSelectorCurvesTest, LoadBalancerConfigYamlTest (all in rtp-proxy-common). Existing WeightedAverageBackendSelectorTest, BackendSelectorRegionFilteringTest, BackendSelectorServerIdFilterTest remain green via the legacy-weight synthesis path.
Region-Pair Scoring (2026-05-22)¶
The original ADR scored backends. In practice the no-region /rtp entry point (blank command on a lobby; proxy-side join with no regionKey hint) needs (serverId, regionKey), not just serverId. Before this amendment that decision lived in a separate "v1 most-kept" picker (PeerRegionRegistry.pickMostKept, hardcoded "max regionKeptCounts wins, lex tiebreak"). That forked the load-balancing logic across lobby and proxy.
RegionAwareSelector unifies both call sites on one scoring path:
- The candidate set is every
(b, r)pair surfaced by the snapshot (b.regions()falling back toregionsAvailablefalling back to the keys ofregionKeptCounts). - Filtering matches the backend-scoped selector: staleness,
pluginState == READY,acceptingRequests,!killSwitch, optionalexcludedServerId(lobby self), optionalworldKeyconstraint. A pinnedrequest.regionKeycollapses the per-backend region loop to that single key. - Scoring:
score(b, r) = ( Σ_i weight_i * curve_i( input_i.normalize(b, r) ) ) / backendWeight(b). TheΣ_iruns overLoadBalancerConfig.terms()plus, when no explicitKEPT_REGIONterm is present andregionScarcityWeight > 0, the synthesized region-scarcity term. - Tie-break:
scoreasc,serverIdasc,regionKeyasc.
Synthesized Region-Scarcity Term¶
A new MetricInput.KEPT_REGION extracts 1 - min(1, regionKeptCounts[r] / 64). Operators may add it as an explicit ScoringTerm and pick any curve. When they don't, the config synthesizes one with weight regionScarcityWeight (default 1.0) and curve exponential(k=5). The exponential default is deliberate: a region 90% empty contributes ~0.85 to the score while a region only 50% empty contributes ~0.07, "drastically prioritising warm regions when shallow". An operator who wants a different shape adds an explicit KEPT_REGION term and the synthesized one steps aside (one knob per axis; no special-case curve config field, every input gets its curve through ScoringTerm like all the others).
Where The Selector Is Used¶
- Lobby-side (
PeerRegionRegistry.pickMostKept): the blank/rtpentry from a lobby. ExcludeslocalServerId(so the lobby doesn't pick itself), applies the existinglocalDecrementsbookkeeping via thePostScoreAdjusthook. - Proxy-side fallback: when
RtpDispatcherreceives anRtpRequestwith noregionKey(e.g. a join-time RTP that the lobby did not pre-decide), the dispatcher callsRegionAwareSelectorto produce(serverId, regionKey)and uses both for the reservation. The existingWeightedAverageBackendSelector.choosepath remains the entry point for requests that already carry aregionKeyor aserverIdFilter; the two coexist by design.
Region == Server Collapse¶
In typical deployments each backend hosts one region (often named after the server). The pair-wise sum collapses to "per-backend score plus a constant", and the (serverId asc, regionKey asc) tiebreak preserves the determinism of the legacy backend-only path. Multi-region backends are handled the same way: each (b, r) is scored independently, so a backend with two regions does not unfairly outweigh a peer with one.
Why Not Fork Lobby And Proxy¶
Operators tune curves and weights in network.yml once. The lobby and proxy disagree only when they observe different snapshots (different Redis read instants), which self-corrects on the next heartbeat tick. Forking would mean two config blocks, two tuning passes, and inevitable divergence.
Coverage¶
RegionAwareSelectorTest (12 cases): defaults pick the warmest region; empty-pool penalty under the exponential default; pinned regionKey collapse; self-exclusion; killSwitch/STARTING/draining filter; staleness filter; MSPT term overriding scarcity when weighted high; explicit KEPT_REGION term suppressing the synthesized one; regionScarcityWeight=0 disabling the bias; per-(backend, region) pair-wise scoring; backendWeights divisor; PostScoreAdjust hook (the lobby decrements analog). LobbyModeTest (18 cases) covers the lobby integration path through PeerRegionRegistry.pickMostKept and remains green byte-identical to the legacy "most-kept" behaviour with default config.
References¶
- ADR-036 Load Balancer section.
REQ-RTP-NET-010(configurable LB policy).REQ-RTP-PROXY-COMMON-002…004,-008.