Skip to content

corpus-atlas-ADR-001 - Scope-Preserving AST Synthesis, Dual-Regime Small-LLM Grounding, and Zero-Open Search Cards

Status: Accepted Date: 2026-10-27

Context

corpus-atlas/build_doc_index.py and scripts/repo_ast.py provide AST and documentation indexing across the repository. While repo_ast_search and repo_ast_context achieved 100.0% recall on literal identifier stems (Front 4) and Gov(c) invariants (Front 2), empirical testing on 12 practice, synonym, and symptom queries (Front 6) exposed two remaining failure modes:

  1. Practice & Synonym Blindness (50.0% Top-1 / 58.3% Top-3 Recall):
  2. When an agent or developer searches by engineering practice, algorithm, or operational symptom ("rate limiter token bucket cooldown", "idempotent close double release chunk ticket", "folia region thread ownership check handoff", "spatial bitmask coordinate claim"), deterministic camelCase splitting and Porter stemming fail when the target class is named after its mathematical or domain model (BudgetPacer, MemoryTracker, FoliaSchedulerImpl, MemoryShape, RTPAPI).
  3. Flat File-Level Bag-of-Words & Forced Follow-Up open Calls (0.0% Zero-Open Sufficiency):
  4. _extract_java_methods_and_summary previously extracted bare method names without parameter types, return types, constructors, or per-method scope boundaries, and flattened all tokens across a .java file into a single bag.
  5. Because search results omitted typed signatures ((ParamTypes) -> ReturnType) and behavioral contracts, agents had to execute a follow-up open call (~3,200 B per file) to verify whether a match was a false conjunction across disjoint methods or to construct a valid call site.
  6. Why Standard Vector RAG / Unconstrained LLM Summarization Loses Precision:
  7. Unconstrained LLM summaries paraphrase away exact type names, numeric bounds, and thread-ownership guards, or blend speculative buzzwords into the same scoring tier as literal AST symbols.
  8. In the dual-regime corpus-atlas/corpus_atlas.py engine (TECHNICAL_SCHEMA for LeafRTP and DOCUMENT_PROSE for oWoD) and discord-support-bot, unconstrained prose synthesis also risks violating split-plane isolation (REQ-DISC-003) or flattening unreliable in-character narrators (IC_NARRATOR, EPISTOLARY_POST) and edition retcons (V20 vs V5) into unattributed facts.

Decision

Split code and prose indexing into a Tier-1 Deterministic Scope Skeleton (exact structural truth) and a Tier-2 Grounded Synthesis Layer (code_ast_synthesis and regime-aware scan_section_with_small_llm), governed by a < 5 ms deterministic AST verification gate and asymmetric search scoring.

1. Tier-1 Brace-Balanced Method Scope Extraction (corpus-atlas/build_doc_index.py)

  • _extract_java_method_scopes and _extract_java_ast_synthesis parse brace-balanced constructor and method scopes (JAVA_CTOR_SCOPE_RE, JAVA_METHOD_SCOPE_RE), recording:
  • signatures_json: Up to 10 compact typed entry-point signatures (MethodName(ParamType name, ...) -> ReturnType), prioritizing public/protected factories and entry points ahead of private helpers.
  • scope_facets_json: Per-method call targets (Collaborator.method), referenced fields, config_refs, thrown exceptions, and concurrency primitives (CAS, Atomic, CompletableFuture, isOwnedByCurrentRegion, FoliaRegionScheduler, Lock, synchronized).
  • Deterministic Structural Practices (practice_tags, synonym_intents, investigate_when, behavior_summary): Maps structural AST idioms directly to canonical hyphenated tags and a <= 220 character behavioral micro-spec (source_mode='DETERMINISTIC_AST').
  • Detect @interface declarations as kind='ANNOTATION' in code_class_meta so marker annotations are de-boosted (0.55x) relative to executable SPIs during capability search.

2. Content-Addressed code_ast_synthesis Table & < 5 ms Incremental Sync

  • Store synthesized facets in a dedicated code_ast_synthesis table keyed by (path, project) and ast_sha256 (SHA-256 of comment-stripped normalized AST tokens), strictly isolated from public doc_sections (REQ-DISC-003).
  • During build_index and sync_dirty_files, existing rows with source_mode='LLM_VERIFIED' and a matching ast_sha256 are preserved untouched; modified files immediately refresh their DETERMINISTIC_AST row in < 5 ms and become eligible for background P2 re-synthesis.

3. Tier-Aware 4-Stage bamboo Synthesis Funnel (SynthesizeAstContextHandler in scripts/local_llm_queue.py)

  • Add task_type="synthesize-ast-context" (P2 priority) ordered top-down by ADR-115 DAG tier (TIER_1_SPI_PURE -> TIER_2_CORE_DOMAIN -> TIER_3_PLATFORM_ADAPTER -> TIER_4_LEAF_ENTRY), injecting 1-line summaries of imported Tier 1/2 SPIs when prompting for Tier 3/4 classes:
  • Stage 1 (Forward Synthesis): Uses RTP_LLM_SYNTH_MODEL (defaulting to qwen2.5-coder:32b on bamboo) with tier-specific prompt lenses (reuse/algorithm synonyms for Tiers 1-2; native server APIs, region thread handoffs, and ticket lifecycle hazards for Tier 3; command routing, S-007 busy gates, and cancel/refund flows for Tier 4).
  • Stage 2 (< 5 ms Deterministic AST Grounding Gate): verify_ast_synthesis_candidate verifies that every camelCase() method call and ADR-*/REQ-*/S-00x ID cited in behavior_summary exists in the class's deterministic AST symbol set, blocks concurrency hallucinations (e.g., claiming lock-free or non-blocking when synchronized is present), strips generic stopwords, and enforces token regexes (^[a-z0-9_.-]{3,64}$).
  • Stage 3 (Asymmetric Entailment Critic): Uses a distinct model family (RTP_LLM_SYNTH_CRITIC_MODEL, defaulting to deepseek-r1:8b on bamboo) to check semantic entailment against the deterministic signatures and scope facets (REQ-LLM-Q-003).
  • Stage 4 (1-Pass Bounded Repair or Deterministic Fallback): Repairs only flagged fields or falls back cleanly to DETERMINISTIC_AST.

4. Asymmetric Scoring & 3-Line Zero-Open Cards (scripts/repo_ast.py)

  • In query_search, Tier-1 exact AST matches remain dominant (class_name 30.0 * idf, camelCase stem 14.0 * idf, method_sig 8.0 * idf, interfaces 7.5 * idf), while Tier-2 facets score at bounded expansion weights (practice_tags / synonym_intents 6.5 * idf, investigate_when 5.5 * idf, behavior_summary 4.0 * idf) with a 1.28x same-scope conjunction bonus when query tokens co-occur across both Tier-1 AST and Tier-2 practice facets or across >= 2 distinct synthesis facets.
  • format_search_markdown (for Top-3 hits) and format_context_markdown render a 3-line card (Signatures, Behavior, Practices / Intents) so calling agents can verify relevance and write call sites without calling open.

5. Helper Bot & oWoD (DOCUMENT_PROSE) Dual-Regime Integration

  • Helper Bot (discord-support-bot / build_doc_index.py:search_code_sections): Enriches second-class code diagnostic hits with method-scoped config_refs, practice_tags, investigate_when, and behavior_summary while keeping code_ast_synthesis strictly out of public doc_sections (REQ-DISC-003).
  • oWoD (corpus-atlas/corpus_atlas.py:scan_section_with_small_llm & bake_project_ast): Generalizes section synthesis across both TECHNICAL_SCHEMA and DOCUMENT_PROSE, extracting implicit entity claims (DIABLERIZED, TRUE_IDENTITY_OF, SIRED_OR_CREATED, DESTROYED_OR_PURGED, ALLIED_OR_JOINED, FOUGHT_OR_OPPOSED, VIEWS_OR_JUDGES) and EPITHET_ALIAS candidate refs when pronoun or clause distance exceeds the 130-char regex window, gated by deterministic symbol-ID lookup and verbatim quote-token grounding while stamping voice, perspective_lens, edition, and printed_p deterministically from section metadata.

Alternatives Considered

Alternative Why Rejected
Dense vector embeddings over raw .java and .md chunks Loses exact symbol boundaries, suffers from cross-method scope bleed, cannot enforce < 5 ms AST symbol grounding, and dilutes Gov(c) invariant precision.
Flattening LLM-generated keywords into code_class_meta.methods / javadoc_summary Overwrites deterministic AST ground truth, prevents ast_sha256 selective invalidation, and allows synonym matches to outrank literal symbol definitions.
Running a single heavyweight reasoning model (deepseek-r1:32b) for both Forward and Critic Violates asymmetric model diversity (REQ-LLM-Q-003) and wastes <think> tokens on forward code-to-JSON extraction where qwen2.5-coder:32b + deepseek-r1:8b critic is faster and higher precision.
Identical synthesis prompt across all 4 DAG tiers Causes Tier 3 platform adapters and Tier 4 leaf commands to flood the synonym index with generic terms (teleport, player, world), diluting Tier 1/2 SPI anti-redundancy ranking.

Consequences

  • Positive:
  • Empirical Front 6 Improvement (bench_repo_ast.py --gate): Practice/synonym Top-3 recall rose from 58.3% (7/12) to 100.0% (12/12), Top-1 recall rose from 50.0% (6/12) to 91.7% (11/12), and Zero-Open Call-Site Sufficiency rose from 0.0% (0/12) to 100.0% (12/12), avoiding 12/12 follow-up open calls (~38,400 B saved across the 12 benchmark tasks).
  • Zero Regression on Fronts 1-5: Query p95 (42.90 ms < 50.0 ms), single-file dirty sync p95 (2.34 ms < 150.0 ms), literal SPI discovery (100.0%), Gov(c) recall (100.0%), and REQ-DISC-003 split-plane isolation (0 leaks) all pass GO_PROMOTE.
  • Offline & CI Determinism: When bamboo is unreachable, build_doc_index.py deterministically populates code_ast_synthesis (source_mode='DETERMINISTIC_AST'), guaranteeing reproducible offline builds.
  • Negative / Trade-offs:
  • Adds one SQLite table (code_ast_synthesis) and ~220 B per Top-3 search hit for the 3-line signature/behavior card.
  • _ROW_TOKEN_CACHE in scripts/repo_ast.py is required to keep warm corpus tokenization under < 3 ms after dirty-file syncs.

References

  • ADR-115 (docs/adr/ADR-115-package-dag-disentanglement-and-decision-preservation-hypergraph.md)
  • docs/dev/AI_USAGE.md (REQ-LLM-Q-001..003)
  • corpus-atlas/build_doc_index.py (CodeAstSynthesisRow, _extract_java_method_scopes, _extract_java_ast_synthesis)
  • corpus-atlas/corpus_atlas.py (scan_section_with_small_llm, bake_project_ast)
  • scripts/local_llm_queue.py (SynthesizeAstContextHandler)
  • scripts/repo_ast.py & scripts/bench_repo_ast.py (benchmark_front6_practice_synonym_and_zero_open)