corpus-atlas-ADR-001 - Scope-Preserving AST Synthesis, Dual-Regime Small-LLM Grounding, and Zero-Open Search Cards¶
Status: Accepted Date: 2026-10-27
Context¶
corpus-atlas/build_doc_index.py and scripts/repo_ast.py provide AST and documentation indexing across the repository. While repo_ast_search and repo_ast_context achieved 100.0% recall on literal identifier stems (Front 4) and Gov(c) invariants (Front 2), empirical testing on 12 practice, synonym, and symptom queries (Front 6) exposed two remaining failure modes:
- Practice & Synonym Blindness (
50.0%Top-1 /58.3%Top-3 Recall): - When an agent or developer searches by engineering practice, algorithm, or operational symptom (
"rate limiter token bucket cooldown","idempotent close double release chunk ticket","folia region thread ownership check handoff","spatial bitmask coordinate claim"), deterministic camelCase splitting and Porter stemming fail when the target class is named after its mathematical or domain model (BudgetPacer,MemoryTracker,FoliaSchedulerImpl,MemoryShape,RTPAPI). - Flat File-Level Bag-of-Words & Forced Follow-Up
openCalls (0.0%Zero-Open Sufficiency): _extract_java_methods_and_summarypreviously extracted bare method names without parameter types, return types, constructors, or per-method scope boundaries, and flattened all tokens across a.javafile into a single bag.- Because search results omitted typed signatures (
(ParamTypes) -> ReturnType) and behavioral contracts, agents had to execute a follow-upopencall (~3,200 Bper file) to verify whether a match was a false conjunction across disjoint methods or to construct a valid call site. - Why Standard Vector RAG / Unconstrained LLM Summarization Loses Precision:
- Unconstrained LLM summaries paraphrase away exact type names, numeric bounds, and thread-ownership guards, or blend speculative buzzwords into the same scoring tier as literal AST symbols.
- In the dual-regime
corpus-atlas/corpus_atlas.pyengine (TECHNICAL_SCHEMAforLeafRTPandDOCUMENT_PROSEforoWoD) anddiscord-support-bot, unconstrained prose synthesis also risks violating split-plane isolation (REQ-DISC-003) or flattening unreliable in-character narrators (IC_NARRATOR,EPISTOLARY_POST) and edition retcons (V20vsV5) into unattributed facts.
Decision¶
Split code and prose indexing into a Tier-1 Deterministic Scope Skeleton (exact structural truth) and a Tier-2 Grounded Synthesis Layer (code_ast_synthesis and regime-aware scan_section_with_small_llm), governed by a < 5 ms deterministic AST verification gate and asymmetric search scoring.
1. Tier-1 Brace-Balanced Method Scope Extraction (corpus-atlas/build_doc_index.py)¶
_extract_java_method_scopesand_extract_java_ast_synthesisparse brace-balanced constructor and method scopes (JAVA_CTOR_SCOPE_RE,JAVA_METHOD_SCOPE_RE), recording:signatures_json: Up to 10 compact typed entry-point signatures (MethodName(ParamType name, ...) -> ReturnType), prioritizing public/protected factories and entry points ahead of private helpers.scope_facets_json: Per-method call targets (Collaborator.method), referenced fields,config_refs, thrown exceptions, and concurrency primitives (CAS,Atomic,CompletableFuture,isOwnedByCurrentRegion,FoliaRegionScheduler,Lock,synchronized).- Deterministic Structural Practices (
practice_tags,synonym_intents,investigate_when,behavior_summary): Maps structural AST idioms directly to canonical hyphenated tags and a<= 220character behavioral micro-spec (source_mode='DETERMINISTIC_AST'). - Detect
@interfacedeclarations askind='ANNOTATION'incode_class_metaso marker annotations are de-boosted (0.55x) relative to executable SPIs during capability search.
2. Content-Addressed code_ast_synthesis Table & < 5 ms Incremental Sync¶
- Store synthesized facets in a dedicated
code_ast_synthesistable keyed by(path, project)andast_sha256(SHA-256 of comment-stripped normalized AST tokens), strictly isolated from publicdoc_sections(REQ-DISC-003). - During
build_indexandsync_dirty_files, existing rows withsource_mode='LLM_VERIFIED'and a matchingast_sha256are preserved untouched; modified files immediately refresh theirDETERMINISTIC_ASTrow in< 5 msand become eligible for backgroundP2re-synthesis.
3. Tier-Aware 4-Stage bamboo Synthesis Funnel (SynthesizeAstContextHandler in scripts/local_llm_queue.py)¶
- Add
task_type="synthesize-ast-context"(P2priority) ordered top-down byADR-115DAG tier (TIER_1_SPI_PURE->TIER_2_CORE_DOMAIN->TIER_3_PLATFORM_ADAPTER->TIER_4_LEAF_ENTRY), injecting 1-line summaries of imported Tier 1/2 SPIs when prompting for Tier 3/4 classes: - Stage 1 (Forward Synthesis): Uses
RTP_LLM_SYNTH_MODEL(defaulting toqwen2.5-coder:32bonbamboo) with tier-specific prompt lenses (reuse/algorithm synonyms for Tiers 1-2; native server APIs, region thread handoffs, and ticket lifecycle hazards for Tier 3; command routing,S-007busy gates, and cancel/refund flows for Tier 4). - Stage 2 (
< 5 msDeterministic AST Grounding Gate):verify_ast_synthesis_candidateverifies that everycamelCase()method call andADR-*/REQ-*/S-00xID cited inbehavior_summaryexists in the class's deterministic AST symbol set, blocks concurrency hallucinations (e.g., claiminglock-freeornon-blockingwhensynchronizedis present), strips generic stopwords, and enforces token regexes (^[a-z0-9_.-]{3,64}$). - Stage 3 (Asymmetric Entailment Critic): Uses a distinct model family (
RTP_LLM_SYNTH_CRITIC_MODEL, defaulting todeepseek-r1:8bonbamboo) to check semantic entailment against the deterministic signatures and scope facets (REQ-LLM-Q-003). - Stage 4 (1-Pass Bounded Repair or Deterministic Fallback): Repairs only flagged fields or falls back cleanly to
DETERMINISTIC_AST.
4. Asymmetric Scoring & 3-Line Zero-Open Cards (scripts/repo_ast.py)¶
- In
query_search, Tier-1 exact AST matches remain dominant (class_name30.0 * idf, camelCase stem14.0 * idf,method_sig8.0 * idf,interfaces7.5 * idf), while Tier-2 facets score at bounded expansion weights (practice_tags/synonym_intents6.5 * idf,investigate_when5.5 * idf,behavior_summary4.0 * idf) with a1.28xsame-scope conjunction bonus when query tokens co-occur across both Tier-1 AST and Tier-2 practice facets or across>= 2distinct synthesis facets. format_search_markdown(for Top-3 hits) andformat_context_markdownrender a 3-line card (Signatures,Behavior,Practices / Intents) so calling agents can verify relevance and write call sites without callingopen.
5. Helper Bot & oWoD (DOCUMENT_PROSE) Dual-Regime Integration¶
- Helper Bot (
discord-support-bot/build_doc_index.py:search_code_sections): Enriches second-class code diagnostic hits with method-scopedconfig_refs,practice_tags,investigate_when, andbehavior_summarywhile keepingcode_ast_synthesisstrictly out of publicdoc_sections(REQ-DISC-003). oWoD(corpus-atlas/corpus_atlas.py:scan_section_with_small_llm&bake_project_ast): Generalizes section synthesis across bothTECHNICAL_SCHEMAandDOCUMENT_PROSE, extracting implicit entity claims (DIABLERIZED,TRUE_IDENTITY_OF,SIRED_OR_CREATED,DESTROYED_OR_PURGED,ALLIED_OR_JOINED,FOUGHT_OR_OPPOSED,VIEWS_OR_JUDGES) andEPITHET_ALIAScandidate refs when pronoun or clause distance exceeds the 130-char regex window, gated by deterministic symbol-ID lookup and verbatim quote-token grounding while stampingvoice,perspective_lens,edition, andprinted_pdeterministically from section metadata.
Alternatives Considered¶
| Alternative | Why Rejected |
|---|---|
Dense vector embeddings over raw .java and .md chunks |
Loses exact symbol boundaries, suffers from cross-method scope bleed, cannot enforce < 5 ms AST symbol grounding, and dilutes Gov(c) invariant precision. |
Flattening LLM-generated keywords into code_class_meta.methods / javadoc_summary |
Overwrites deterministic AST ground truth, prevents ast_sha256 selective invalidation, and allows synonym matches to outrank literal symbol definitions. |
Running a single heavyweight reasoning model (deepseek-r1:32b) for both Forward and Critic |
Violates asymmetric model diversity (REQ-LLM-Q-003) and wastes <think> tokens on forward code-to-JSON extraction where qwen2.5-coder:32b + deepseek-r1:8b critic is faster and higher precision. |
| Identical synthesis prompt across all 4 DAG tiers | Causes Tier 3 platform adapters and Tier 4 leaf commands to flood the synonym index with generic terms (teleport, player, world), diluting Tier 1/2 SPI anti-redundancy ranking. |
Consequences¶
- Positive:
- Empirical Front 6 Improvement (
bench_repo_ast.py --gate): Practice/synonym Top-3 recall rose from58.3%(7/12) to100.0%(12/12), Top-1 recall rose from50.0%(6/12) to91.7%(11/12), and Zero-Open Call-Site Sufficiency rose from0.0%(0/12) to100.0%(12/12), avoiding12/12follow-upopencalls (~38,400 Bsaved across the 12 benchmark tasks). - Zero Regression on Fronts 1-5: Query p95 (
42.90 ms < 50.0 ms), single-file dirty sync p95 (2.34 ms < 150.0 ms), literal SPI discovery (100.0%),Gov(c)recall (100.0%), andREQ-DISC-003split-plane isolation (0leaks) all passGO_PROMOTE. - Offline & CI Determinism: When
bamboois unreachable,build_doc_index.pydeterministically populatescode_ast_synthesis(source_mode='DETERMINISTIC_AST'), guaranteeing reproducible offline builds. - Negative / Trade-offs:
- Adds one SQLite table (
code_ast_synthesis) and~220 Bper Top-3 search hit for the 3-line signature/behavior card. _ROW_TOKEN_CACHEinscripts/repo_ast.pyis required to keep warm corpus tokenization under< 3 msafter dirty-file syncs.
References¶
ADR-115(docs/adr/ADR-115-package-dag-disentanglement-and-decision-preservation-hypergraph.md)docs/dev/AI_USAGE.md(REQ-LLM-Q-001..003)corpus-atlas/build_doc_index.py(CodeAstSynthesisRow,_extract_java_method_scopes,_extract_java_ast_synthesis)corpus-atlas/corpus_atlas.py(scan_section_with_small_llm,bake_project_ast)scripts/local_llm_queue.py(SynthesizeAstContextHandler)scripts/repo_ast.py&scripts/bench_repo_ast.py(benchmark_front6_practice_synonym_and_zero_open)