Skip to content

AI Usage in RTP Development

This is a writeup of how and why AI tooling is used on RTP. It's for two audiences: skeptics who want to know whether I'm using AI responsibly, and younger devs trying to figure out what AI is and isn't good for. I'm not selling anything. Where AI does poorly, that's on the page too.


A. The "why"

My background is academic. The part that matters here isn't the credential, it's that I studied this in college and designed tensor processing units alongside peers - modeled hardware and statistical models for machine learning. So when I look at an LLM I'm not looking at magic. I'm looking at compression and retrieval over an n-dimensional table (one that could run a lot leaner than the way it's currently served), and I have a pretty good idea what it's doing and what it isn't.

LLMs are designed to sound like the people who produced them. The training corpus is human-written text and the objective is to produce text another human would find plausible. They talk like me only because I'm one of the humans in the pile. More accurately they talk like the aggregate of every author whose work landed in the training set, weighted by how the data was curated.

Plausible isn't the same as correct. When the model is asked something the training didn't cover, it doesn't know that. It falls back on the closest neighboring memory and produces something confident. That's basically what a human does while dreaming: yesterday's fragments restitched. We call them hallucinations for good reason.

Some models, including the one Junie runs on, are tuned to hold lower confidence and to ask before acting. That's what makes them usable for the kind of work I describe below. Without it you've got a confident stranger typing fast.

I brought AI into RTP for two reasons and neither of them is the reason it usually gets marketed on.

First, repetitive wrappers. RTP's V3 architecture is modular on purpose. A new shape, a new platform adapter, a new safety predicate - these all slot into seams I designed during V2. The interesting work happened when I built the seams. Writing the Nth thing that conforms to them is gruntwork, and gruntwork is exactly the kind of locally-specified problem an LLM can chew through without having to invent any design.

Second, knowledge gaps on specific APIs. I know RTP thoroughly. I don't know Fabric's intermediary mappings or Velocity's plugin SPI to the same depth. The system design for the Fabric and proxy frontiers is mine. The specific method signatures, the obf/unobf carrier split, the Velocity event class names - those aren't things I'd otherwise pick up in the timeframe I had.

The timeframe is a real part of this. I was in the middle of a personal move during V3 work and didn't have time to burn on extraneous gruntwork in a system I already know. So the honest answer isn't "AI is the future of programming". It's "I had a list of mechanical tasks and a finite week".


B. The "how"

Since people are using AI now and I needed some raw output, I gave it a try. I added guards (process rules in documentation and AGENTS.md) for LLM-based tooling to ensure human-originated architecture. It was able to help with speed, but watching the diff I saw some issues. I had to pull in specific reference material and establish a "read and document" practice for the tools to hit a viable quality standard, which I maintain based on reviewing diffs.

The ADR workflow and the documentation discipline in this repo (the shall phrasing in REQUIREMENTS.md, the separation of what from how, the per-subproject ADR sequences, the REQ-* traceability matrix) didn't come from AI. That came from my time working on human-safety-related systems, where auditors and post-incident reviewers expect a particular shape of paper trail.

Lucky for me, that same shape happens to be exactly what regulates an LLM well. The format forces explicit what, explicit why, explicit trade-offs, explicit consequences, and numbered so nothing gets quietly retconned. A model told "produce an ADR matching this template, here's the brainstorm, here are the trade-offs we identified" produces useful drafts. A model told "design the system" produces plausible noise.

When AI is left to prioritize on its own it picks badly. It treats whatever was mentioned most recently as the most important thing, regardless of whether it matters at the system level. The fix is what you'd do with a junior dev who just learned a new pattern and is now seeing it everywhere: somebody with the wider context window says "no, that's a footnote, not a chapter".

Refactoring only happens after a human design review and a written checklist. I halt sessions mid-stream more than half the time to fix the model's assumptions and redirect it. That's not me complaining about the tool, that's the intended workflow. The model is tuned to ask, I'm calibrated to interrupt. A session that runs end-to-end without me jumping in is a session I should reread carefully, because either the task was small or I missed a wrong turn. The checklist discipline in .junie/AGENTS.md ("Checklist-Based State Tracking") exists partly for this reason - it's how I make an interrupted session resumable from a verified state instead of from the model's summary of what it thinks it did.

Unit tests are a lower-risk place to let AI write. The inputs, the call, and the expected outputs are all locally specified. A failing test is the artifact that proves I understood the bug before I fixed it. And a bad test is loud - it fails, or it passes trivially in a way someone will notice. A bad refactor is quiet.

The V2 branch was written without AI, mostly because generative AI wasn't really a thing back then. That timing was lucky. The seams V3 plugs into (region pipeline, safety predicates, shape contributors, platform adapters) are mine, designed before I trusted an LLM to participate. AI gets used for wrappers around that human work or repeated code in the same shape. A new Shape that conforms to the existing Shape SPI is a good fit: the contract is fixed, the tests around it already exist, and the architecture-boundary tests (RTPArchitectureTest) fail loudly if a wrapper reaches across a module it shouldn't. AI accelerates the Nth instance of a pattern I already designed. It doesn't get to invent the pattern.

Two pieces of tooling in this repo are worth calling out because they look like human workflows and aren't. The locale TSV pipeline under scripts/out/ is only used by AI. A human translator fluent in the target language can just open lang/<locale>/<file>.yml and edit it, and the TSV layer is unnecessary overhead. The pipeline exists because AI-based translation benefits from a single condensed table of every locale's translation of every key, comments included, with placeholders pre-masked. That's the shape of input an LLM can review without losing track. So the pipeline is tooling I built for AI specifically, with the simpler human path left alone.

The devstack at platforms/proxy/devstack/ is the reverse case. I designed it to help run multi-server integration tests and code reviews. AI didn't design it. AI wrote the Docker Compose YAML, the entrypoint shims, and the reset script after I told it the topology (1 Redis, 2 proxies, 2 lobbies, 3 platform-asymmetric backends). With the introduction of the headless Mineflayer bot driver (ADR-091) and autonomous script flags, devstack acceptance testing is fully scriptable and can be executed by both human contributors and AI agents non-interactively (see docs/dev/TESTING_GUIDE.md).

A third piece of infrastructure is the repository AST and FTS5 knowledge graph under corpus-atlas/ (corpus-atlas/build_doc_index.py, corpus-atlas/corpus_atlas.py, corpus-atlas/mcp_server.py, and corpus-atlas/generated/doc-index.sqlite), exposed to developer workflows through both the persistent stdio MCP server (corpus-atlas/mcp_server.py: repo_ast_context, repo_ast_search, repo_ast_impact, repo_ast_audit, repo_ast_sync, repo_ast_offload) and the CLI (scripts/repo_ast.py), and gated by the 4-front differential benchmark in scripts/bench_repo_ast.py --gate. Instead of spending 5-15 blind grep/open calls orienting on a symbol or reinventing a helper that already exists in api/* or rtp-api, a single < 25 ms warm MCP or CLI call resolves the symbol's ADR-115 decision-preservation neighborhood Gov(c) (governing ADRs, REQ-* traceability rows, S-00x prohibitions, Fan-In TF-IDF weighted callers, 2-hop co-coupled siblings, config keys, and exact Gradle test command) and ranks existing Tier 1/2 SPIs first. Because we control both the search engine and the corpus (Rule D-010 in docs/dev/RULES.md), any rank anomaly in repo_ast.py search or repo_ast.py audit directly flags where a class or ADR is missing a concise capability Javadoc, @governedBy tag, or backticked TRACEABILITY.md reference. To ensure corpus-atlas and the helper-bot retrieval pipeline remain genuinely domain-agnostic rather than overfit to Minecraft plugin YAML, the same zero-dependency SQLite FTS5 + AST engine is validated out-of-tree against an external tabletop rulebook & lore reference setup (DOCUMENT_PROSE regime: wod-lore-atlas + oWoD-rules, spanning 52,000+ indexed pages alongside 4,725 schema-validated 20th-Anniversary rules records across 10 books and 19 sections, 1,210 dice-pool/difficulty rolls, 342 reference tables, 1,361 cross-linked related/parent AST edges, and 40 audited DISC-0001..DISC-0040 text-vs-table discrepancies). None of that external dataset's file set is stored in this repository, but exercising the exact same FTS5, corpus_vocab, and AST conflict-surfacing pipeline across both TECHNICAL_SCHEMA (LeafRTP, TreeGen, OnePlayerSleep) and DOCUMENT_PROSE corpora confirms that the retrieval and verification math generalizes across domains without hardcoded lookup tables.

A fourth piece is the priority-queued local GPU dispatch and staged multi-opinion verification broker (scripts/local_llm_queue.py, backed by scripts/out/llm-queue.sqlite in WAL mode and exposed via repo_ast_offload / python3 scripts/repo_ast.py offload). It offloads high-volume, locally-specified batch work (translate-bundle for bleed-<lang>.tsv, translate-doc for Markdown documentation with <<P#>> code/link sentinel masking, and backfill-javadoc for Rule D-010 @governedBy/@trace Javadocs) as P2 background jobs and runs cold-context adversarial diff audits (critique-diff against Gov(c) ADRs and S-001..S-007 prohibitions) as P1 jobs, while yielding at 5-10 item chunk boundaries whenever a P0 interactive support-bot lease (gpu_leases) or higher-priority job arrives. Every local draft passes through a 4-stage verification funnel (Opinion 1 Forward -> < 5 ms Deterministic Gate -> Opinion 2 Asymmetric Critic -> 1-Pass Bounded Local Repair) so syntactically invalid drafts never waste second-opinion GPU cycles, and only compact summary counts plus unresolved NEEDS_REVIEW items (scripts/out/review-*.md, <= 40 lines) are returned to the remote agent.

A fifth piece is the Discord support bot under discord-support-bot/ (see discord-support-bot/DISCORD_SUPPORT_BOT_DESIGN.md), which builds on that same SQLite FTS5 and AST index and applies the same philosophy to user support: use deterministic code for everything that has to be exact, and put a tight cage around the parts where statistical models help. Most of its 7-stage pipeline avoids LLMs on purpose. Foreign-language posts harvest exact localized YAML keys from .lang.yml files before running two non-LLM translators (Google Translate NMT and MyMemory Translation Memory) scored against the schema, screenshots use non-LLM CPU OCR (RapidOCR ONNX + CTC character recognition with corpus-vocabulary word decompounding), and vague symptom posts with no diagnostics abstain deterministically (0 GPU usage) to ask for /rtp info, /rtp test, or mclo.gs logs. Untrusted user prose never reaches the LLM at all: each clause is projected onto the closed repository vocabulary and sorted alphabetically into an orderless token set ($P(\text{permute}(x)) = P(x)$), erasing imperative syntax so prompt injection cannot cross the boundary.

Where the bot does use AI (LocalLlmSynthesizer, defaulting to local deepseek-r1:32b with zero host-shell or filesystem-write access and optional read-only sandbox verification tools via BotVerificationWorkbench), it is for goal-aligned causal reasoning over retrieved corpus blocks (intended-usage.md, what-not-to-do.md, ADRs, second-class Java diagnostics, and YAML schema hits) - for instance, telling apart "make the background queue refill faster" from "stop server lag" instead of recommending a setting just because its hazard warning contains the word lag, or optionally invoking the sandbox verification workbench (console_log_sandbox, local_webpage_operations, multi_format_screenshotting, mineflayer_protocol_bot) to check cross-key [RTP bit] invariants and Web Editor previews before answering. Every drafted /rtp subcommand, YAML key, # @range number, # @options enum value, and URL is then verified deterministically against doc-index.sqlite by verify_llm_draft_against_schema before posting, falling back to deterministic FTS5 bullets if verification fails or when the bot runs in zero-LLM mode (--llm-endpoint off).

What AI is not used for, as a rule:

  • Code review. A reviewer has to hold the whole change in mind, decide what's missing, and be accountable. An LLM can't be accountable.
  • Automated git operations. Ever. And never "optimizing" human-written logic. The git-safety prohibitions in .junie/AGENTS.md exist because that failure mode has already happened in this repo. It's not theoretical.
  • System design. Design happens in human brainstorming sessions and AI formats the result. The other direction is how you end up with a codebase nobody understands.
  • Agentic troubleshooting. AI gets used for investigation - reading the relevant files, summarizing what they do, listing where a bug might live - but it frequently misattributes the error source. It can advise. It does not jump to execution. Bug fixes are directed by humans and commanded explicitly.
  • Independent system docs. This document is the example. Every paragraph is human-directed prose that an LLM then helped format. The substance is mine. If you see an AI-written ADR or requirement in this repo that drifts from that, file an issue. It's a defect.