ADR-130 - Shallow-AST Storefront Architecture, Heading-Vector Scoping, and Document Salience Governance¶
Status: Accepted Date: 2026-10-28 Extends: ADR-108, ADR-115, ADR-126, corpus-atlas-ADR-001, corpus-atlas-ADR-002
Context¶
The DailyStruggle/RTP monorepo publishes multiple server plugins and addons (LeafRTP, LeafTreeGen, LeafSleep, LeafConfig, LeafSleuth, LeafRTPGuiAddon) across Modrinth, Hangar, SpigotMC, BuiltByBit, and GitHub (ADR-126). Each product compiles its marketplace listings from a single tagged source (docs/publishing/FRONT_PAGE.md) via scripts/release/build_front_pages.py under Rule D-006 (first-person plain engineering voice, concrete numbers, zero sales hype).
However, empirical inspection of how external search engines (Google Search, Gemini AI Overviews, Perplexity, marketplace search) and our own shallow Markdown/HTML AST indexer (corpus-atlas/build_doc_index.py) reconstruct FRONT_PAGE.md exposed a structural failure mode: a plugin designed to work out of the box can be indexed and summarized as a hyper-technical benchmark pitch or have its resolved fuzzer bugs indexed as current product defects.
Combining the Google Content Warehouse API architecture (CompositeDoc, PerDocData, Goldmine, SalientTermsSet), Google's Heading-Vector Passage Ranking patents (US9959315B1, US20160078102A1), mobile-first <details> indexing behavior, and the Hoare precondition prefixing rules proven in corpus-atlas-ADR-001 and corpus-atlas-ADR-002, four concrete indexing mechanisms govern how storefront pages are interpreted:
1. Heading-Vector Ancestor Inheritance (US9959315B1, US20160078102A1)¶
Search engines and RAG chunkers parse HTML and Markdown into a shallow heading hierarchy (<h1>–<h4> / #–####). Every extracted passage inherits an ancestor Heading Vector ([H1 Product Title] > [H2 Section] > [H3 Subheading]) that tells the passage ranker and LLM synthesizer what topic the passage belongs to.
- Failure Mode (Scope Collapse Across
<details>and---): Horizontal rules (---/<hr>), bold paragraphs (**Everyday commands**,**Q: ...**), and<details><summary>/[SPOILER]containers do not open or close a heading scope in standard DOM/Markdown AST splitters. Any text inside or after them inherits the preceding## H2heading until the next## H2appears. - Observed Impact on
LeafRTP: Because Section 6 (Commands, permissions, configuration files,FAQ,Support, 13.7 KB) followed a---separator with no## H2heading, shallow AST rebuilds attached all commands, permissions, config tables, and 14 operator FAQs under## What paying gets you(onbbb-pro) or## Why I made it(on Modrinth, Hangar, Spigot, andbbb-lite). Similarly, unheaded<details>blocks forHow I test it, and what the tests have foundinside## How it workscaused historical fuzzer bugs to be indexed under## How it works.
2. Prefix & First-Sentence Window Salience (PerDocData.bodyWordsToTokensRatioBegin, Goldmine, Hoare Precondition Slot)¶
Google's PerDocData.bodyWordsToTokensRatioBegin and Goldmine passage scorer assign disproportionate weight to the opening tokens of a document and the first sentence following each heading or bold bullet label. This is the prose equivalent of corpus-atlas-ADR-002, where @requires Hoare preconditions must be hoisted ahead of a method body to survive prefix truncation.
- Failure Mode (Defect-First Bullet Leads): When a test-finding bullet leads with a defect label (
- **Main-thread chunk load on arrival (spark).**) and Sentence 1 narrates the pre-fix bug while Sentence 3 states"The fix keeps the ticket...", shallow snippet extractors (extract_doc_snippet) and passage rankers index the historical defect as how the plugin currently works. - Failure Mode (Missing Automation Precondition on
## How it works): Listing internal pipeline stages (spatial memory, region-file pre-filtering, shuffled selection, or marginal MSPT regression) under## How it workswithout an opening sentence stating that those stages run automatically in the background behind standard operator settings leads AI summarizers to tell users they must manually tune the engine.
3. Full-Weight <details> Indexing & Document Salience Skew (CompositeDoc, SalientTermsSet)¶
Under mobile-first indexing, Google indexes text inside <details> / [SPOILER] accordions at 100% document weight when computing SalientTermsSet (what the page is globally about), while de-prioritizing hidden accordion text for primary search snippets.
- Failure Mode (Asymmetric Summarization & "Comparative Hard Sell" Classification): In an earlier pass on
LeafRTP, user-facingFeaturesandHow it workswere trimmed to short summaries linking todocs/site/, while 15.8 KB of benchmark methodology, three competitor comparison tables (naming rival plugins over 30 times), and fuzzer post-mortems were kept inline onFRONT_PAGE.mdinside<details>blocks. As a result, ~70% of the page's body text (~18,000 of ~25,500 chars) consisted of stress-test and systems-audit prose, and only ~21% covered installing, configuring, and using the plugin. To a document classifier or LLM synthesizer, a page that spends 70% of its tokens dissecting benchmarks and rival plugins reads as a hard-sell pitch for systems engineers rather than an accessible server plugin.
4. High-IDF Entity Dominance & Negation Blindness (SalientTermsSet, Webref, BM25)¶
Entity and term-salience models weight high-IDF (rare) technical tokens (space-filling curves, Hilbert, rejection sampling, SCHED_FIFO, Jazzer, RSA-2048 sequence counters, Zstandard decompressor, old-generation maxHeapPercent, SplitMix64, Charikar SimHash) 5–10x higher than everyday operator words (easy, circle, radius, safe, report).
- Failure Mode (Negation Bleed in Setup Passages): Bag-of-words and shallow passage vectors do not invert term co-occurrence under negation. Writing "Q: Is it complicated to set up? A: No. You never configure space-filling curves or math settings" injects
configure+space-filling curvesinto the setup FAQ vector. - Failure Mode (Unqualified Stress-Harness Overrides): Mentioning benchmark-only overrides (such as
period: 1) without a same-sentence guard warning operators to keep shipped defaults on live servers causes RAG extractors to surface torture-test settings as production recommendations.
Decision¶
Every product storefront page (plugins/<Product>/docs/publishing/FRONT_PAGE.md, addon FRONT_PAGE.md files, and product README.md overviews) in the monorepo shall be structured for accurate shallow-AST reconstruction, heading-vector attribution, and usability-first document salience:
1. Invariant 3-Zone / Explicit ## H2 Hierarchy (Zero Orphaned <details> Scopes)¶
Every product FRONT_PAGE.md follows a fixed 3-Zone progression where every functional topic opens its own explicit ## H2 heading:
| Zone | Target Audience | Required H1 / H2 / H3 Sections |
Primary Question Answered |
|---|---|---|---|
| Zone 1: Verified Outcomes & Universal Guarantees (Above the Fold) | 100% of visitors (evaluators, beginners, power users) | # <Name>: <Plain Category>## What it is (category, commands, out-of-the-box default, hero UI image)### The short version (1 headline + 5 outcome bullets; secondary items in More highlights) |
What is this, is it easy and safe to use, and why pick it? |
| Zone 2: Operational On-Ramp & Zero-Friction Setup | Action-oriented server admins ready to deploy or switch | ## Common setups (3 concrete recipes)## Install (3 steps: drop jar, run command, optional 1–2 setting tweak or visual wizard)## Features (thematic summary linking to docs/site/features.md) |
How do I install and configure it in under two minutes? |
| Zone 3: Architectural Proof, Story & Scoped Reference | Skeptics, Folia/network architects, and operators looking up reference tables | ## Performance (1 summary table + 1 chart; link out for raw tables/rig)## How it works (automation precondition lead + 3–4 plain stage bullets)## Testing and quality gates (4 plain verification bullets + link to docs/site/quality-gates.md)## Why I made it (plain problem/evolution narrative)## What paying gets you (kind: promo, BuiltByBit only)## Commands, configuration, and FAQ (explicit H2 wrapping <details> blocks for Commands/Permissions/Config, FAQ, and Support)## Links |
Why should I trust the numbers, and where do I look up commands or answers? |
- Explicit
H2Boundary Invariant:<details>and---shall never be used as top-level section transitions without an explicit## H2heading. Reference<details>blocks (Commands, permissions, configuration files,FAQ,Support) must sit under## Commands, configuration, and FAQso they never inherit## Why I made itor## What paying gets you. Testing and verification summaries must sit under## Testing and quality gatesso they never inherit## How it works.
2. Usability & Safety First in ### The short version (Zone 1 Balance)¶
- Five Balanced Outcome Bullets: The 5 visible bullets in
### The short versionuse the form- [**Adjective**](link): measured fact.and must lead with Ease of use and Safety / Guardrails before raw throughput: - Ease / Setup (
Easy/Zero-config/Visual): Visual web editor (/rtp editor), 60-second setup wizard (/rtp admin setup), or single-key mode (radius/governor.mode). - Safety / Guardrails (
Safe/Contraption-safe): Terrain/hazard verification, claim-plugin integrations, or redstone/bystander immunity. - Speed / Accuracy (
Fast/Accurate): One canonical headline metric (e.g.,5 ms median from /rtp to arrival on Paper,40 TP/s Paper / 180 TP/s Folia). - Portability / Transparency (
Portable/Explainable/Self-verifying): Zero-friction config migration (/rtp config import) or human-readable diagnostic proof. - Platform Coverage (
One jar): Multi-platform support (Paper,Spigot,Folia,Fabric,NeoForge,Velocity). - No Duplicate Benchmark Pitch in Zone 1/2: Do not stack multiple benchmark-throughput bullets in
### The short version, and do not repeat raw benchmark run counts in## Common setups.
3. Hoare Precondition Prefixing for Sections and Bullets (Contract-First Lead Sentences)¶
## How it worksAutomation Precondition: The first sentence under## How it worksmust explicitly state that all stages run automatically in the background behind standard operator settings (e.g.,CIRCLE/SQUAREandradius, ormode: OBSERVE/GOVERN_SAFE) with no manual tuning.- Fix-First / Invariant-First Bullet Leads: Wherever test findings, profiler discoveries, or historical post-mortems are listed (on
FRONT_PAGE.mdor indocs/site/quality-gates.md), the bold label and Sentence 1 must state the verified fix or preserved invariant (- **Player session cleanup (fuzzer, fixed in 3.3.0).** Disconnected player sessions are released immediately...). The historical defect caught by the tool may only appear in Sentence 2+. - Same-Sentence Benchmark Override Guard: Any sentence on
FRONT_PAGE.mdthat mentions a stress-harness config override (such asperiod: 1) must state in the same sentence that it is a benchmark-only setting and that live servers should keep the shipped defaults.
4. Document Salience Budget & Plain Operator Vocabulary¶
- >= 70% Usability, Safety & Operator Reference Salience: Because search engines index
<details>content at 100% document weight, at least 70% of the body character and token weight ofFRONT_PAGE.mdshall be dedicated to product identity, ease of setup, safety, features, and operator reference/FAQ—never drowned out by stress-test methodology or competitor failure tables. - Single-Table Cap in
## Performance:## PerformanceonFRONT_PAGE.mdshall contain at most 1 summary comparison table and 1 visual chart, with links tohelpers/StressTest*/RESULTS.mdandPRE_WRITEUP.mdfor secondary latency tables, per-plugin control adjustments, hardware/kernel scheduling notes (SCHED_FIFO), and extended caveats. - Concise Quality Summary in
## Testing and quality gates:FRONT_PAGE.mdsummarizes verification in 4 plain-language bullets (built-in in-game/rtp testor/sleuth test, automated safety rule suite, stress & memory testing, coverage & API compatibility) and links todocs/site/quality-gates.mdfor the full tool matrix and fuzzer post-mortems. - High-IDF Jargon Quarantine: High-IDF algorithm, cryptographic, compression, kernel, and JVM heap-pool terms (
space-filling curve,Hilbert,rejection sampling,CIRCLE_NORMAL,RSA-2048,Zstandard,old-generation,maxHeapPercent,SCHED_FIFO,Jazzer,Entity.isInWall) shall not appear in Zone 1, Zone 2,## Testing and quality gates,## Why I made it, or## Commands, configuration, and FAQonFRONT_PAGE.md. Link todocs/site/how-it-works.md,docs/site/why.md,docs/site/quality-gates.md, or subproject ADRs for deep derivations. - Positive Setup Surface (Zero Negation Bleed): Setup instructions (
## Install) and setup FAQ answers (Q: Is it complicated to set up?) shall state the positive 1–2 setting surface (CIRCLE,radius: 4096b,centerRadius: 1024b,/rtp editor,/rtp admin setup) and shall never negate internal math or curve jargon.
5. Mechanical Enforcement in build_front_pages.py and corpus-atlas¶
scripts/release/build_front_pages.pyAST Gate (check_shallow_ast_sections):- Parses every rendered storefront target into shallow AST sections (
H1/H2/H3/<summary>) and fails--check(and CI release workflows) if:### The short versionlacks ease-of-use (Easy) or safety (Safe) bullets.## Performanceexceeds 1 Markdown table or omits a same-sentence benchmark-only guard onperiod: 1.## How it workslacks an opening automation precondition sentence.Commands, permissions, configuration files,FAQ, orSupport<details>blocks inherit## Why I made it,## What paying gets you, or## How it works.- Any finding bullet leads Sentence 1 with a historical defect before stating the fix/invariant.
- Any setup FAQ answer pairs setup/configuration with negated curve/spiral jargon.
- Any banned high-IDF systems/fuzzer/crypto/GC-pool jargon appears on
FRONT_PAGE.md.
corpus-atlas/build_doc_index.pyShallow-AST Parity:split_markdown_sections()splits<summary>blocks into child passages tagged with their parentH2/H3heading vector (Parent Heading > Summary Title), andextract_doc_snippet()strips structural HTML tags (<div>,<details>,<summary>) and---separators before extracting lead sentences.
Alternatives Considered¶
| Alternative | Why Rejected |
|---|---|
Keep 15+ KB of secondary benchmark tables, competitor config audits, and fuzzer post-mortems inline inside <details> blocks on FRONT_PAGE.md |
Google indexes <details> content at 100% weight for document-level SalientTermsSet classification; keeping 70% of the page's tokens on benchmarks and fuzzer internals causes search engines and AI Overviews to classify the product as a hyper-technical hard sell. |
| Delete technical proofs, fuzzer post-mortems, and competitor controls entirely | Violates Rule D-006 and Zone 3 transparency. Moving secondary tables and post-mortems into docs/site/quality-gates.md and helpers/StressTestRTP/RESULTS.md preserves 100% of the engineering evidence one click away without distorting the front-page index profile. |
| Use negated disclaimers in setup FAQs ("You never configure space-filling curves") to reassure beginners | Bag-of-words and shallow passage embeddings do not invert co-occurrence edges under negation; mentioning space-filling curves in a setup answer associates setup with complex math. |
Rely on manual copy review without mechanical AST checks in build_front_pages.py |
Heading scope collapse (--- vs ## H2), defect-first bullet leads, and high-IDF jargon drift back in over iterative edits unless enforced by build_front_pages.py --check and unit tests. |
Consequences¶
- Positive:
- External search engines, AI Overviews, and
corpus-atlasreconstruct every section ofFRONT_PAGE.mdunder its true topic heading (## Testing and quality gates,## Commands, configuration, and FAQ) with zero orphaned<details>blocks. - Document-level term salience shifts from ~70% stress-test/fuzzer internals to >70% usability, setup, safety, and operator reference while keeping 100% of the benchmark and fuzzer evidence linked in
docs/site/andhelpers/StressTestRTP/. - Every new product in the monorepo (
LeafTreeGen,LeafSleep,LeafConfig,LeafSleuth) inherits a repeatable, mechanically linted blueprint for storefront pages that highlights ease of use and safety first. - Negative / Trade-offs:
- Deep benchmark methodology tables and fuzzer bug post-mortems live in
helpers/StressTestRTP/RESULTS.mdanddocs/site/quality-gates.mdrather than inline onFRONT_PAGE.md, requiring one click for readers who want the full raw tables. - Authors adding new sections or FAQs to
FRONT_PAGE.mdmust satisfycheck_shallow_ast_sectionsinscripts/release/build_front_pages.py.
References¶
plugins/LeafRTP/docs/publishing/FRONT_PAGE.mdplugins/LeafRTP/docs/site/quality-gates.md,plugins/LeafRTP/docs/site/how-it-works.md,plugins/LeafRTP/docs/site/features.md,plugins/LeafRTP/docs/site/why.mdplugins/LeafSleuth/docs/publishing/FRONT_PAGE.md,plugins/LeafRTP/addons/LeafRTPGuiAddon/FRONT_PAGE.mdscripts/release/build_front_pages.py(parse_shallow_ast,check_shallow_ast_sections,check_zone1_no_jargon,check_zone2_onramp)scripts/tests/test_build_front_pages.pycorpus-atlas/build_doc_index.py(split_markdown_sections,extract_doc_snippet)docs/dev/RULES.md(Rule D-006,Rule D-010)- corpus-atlas-ADR-001, corpus-atlas-ADR-002
- Google Heading-Vector & Passage Indexing References:
US9959315B1(Generating a heading vector for a document),US20160078102A1(Document scoring based on document content update), and Google Content Warehouse API (CompositeDoc,PerDocData.bodyWordsToTokensRatioBegin,Goldmine,SalientTermsSet).