Does Big Indexer actually help AI coding assistants?

† 79.9% is the all-prompt mean for the refresh batch; 96% is the p04 (safe implementation path) slice across 5 repos. The harder p01–p03 prompts pull the mean down; p04 evidence is the most actionability-relevant slice.

Replication interpretation: actionability, boundary accuracy (≥0.95), and zero-hallucination held across all three models. Evidence coverage varies by model tagging style — see the callout below. Gemini auto's 0.95 boundary reflects one genuine architectural miss (django/p02, depth-first on query.py, not pattern calibration).

Raw-output comparison (same prompt, two models)

Prompt: fastapi p03 — "blast radius for solve_dependencies"

deepseek (opencode_mcp_p03_twin_refresh_r2):
| Definition spans lines 598–735 ... | VERIFIED |
| get_request_handler ...             | VERIFIED |
| get_websocket_app ...               | VERIFIED |
VERIFIED: Only 3 call sites exist in the codebase.

GPT-4o (opencode_mcp_p03_twin_refresh_gpt4o_r1):
1. VERIFIED: Changes to solve_dependencies will directly impact tests.
2. HYPOTHESIS: Other clusters may have dependent implications.

Source artifacts: deepseek run · GPT-4o run

BGI TWIN rows are from the post-shipment refresh batch (p01–p04, fresh runs, explicit twin_context invocation, CallToolRequest evidence in every run). Boundary accuracy is 0–1 per run (correct seam identification). Actionability is 1–5.

Self-reported scoring

Ground-truth checklists were written by us and scored by us. The checklists were defined by reading the actual source code before scoring (not after). The full scoring rubric is at validation/SCORING_RUBRIC.md. Every raw AI output is committed to validation/runs/ on GitHub. Anyone can re-score independently. If you score any run differently, open an issue with your reasoning — we will update the record.

100 runs across 5 repos is stronger, but still limited

Python + Go + TypeScript is better than Python-only, and we now have three-model independent replication (deepseek, GPT-4o, Gemini auto), but 5 repos is still a small sample. The pydantic-core finding (0% → 80% evidence on a Rust/Python hybrid) is strong enough to stand independently. For stronger statistical confidence, we still need more repos and at least one external team replication.

BGI-TWIN refresh is MCP-only — no updated baseline

The post-shipment refresh only ran the MCP+TWIN mode. The baseline numbers in the per-repo table are from the earlier A/B batch. We have no reason to believe the baseline changed (same repos, same prompts), but this is a real limitation of the experimental design.

Evidence coverage is sensitive to explicit tagging style

This rubric rewards explicit claim-level VERIFIED/HYPOTHESIS/UNKNOWN tagging with file/line citations. deepseek follows this protocol most strictly (13.9 tags/run), Gemini auto is intermediate (8–15/run), and GPT-4o tags less explicitly (6.95/run). All three produce correct answers with high actionability and zero hallucinations — the coverage gap is a rubric/style interaction, not a quality gap. The tag-relaxed second score accounts for this.

One MCP run did not invoke tools

One next.js p04 run in the original A/B batch was confirmed as invalid (no CallToolRequest in output) and is explicitly marked in runs.csv as unscored. All 60 BGI-TWIN refresh runs (deepseek, GPT-4o, Gemini) have CallToolRequest evidence present.

We still need external replication

We completed three-model independent replication (GPT-4o and Gemini auto). The next credibility jump is an external replication: one engineer who has never used Big Indexer running this protocol on their own repo and publishing raw outputs. If you are willing to do this, open an issue and we will send the exact protocol.

# 1. Install
pip install bigindexer

# 2. Clone any repo and build the BGI index
git clone --depth 1 https://github.com/tiangolo/fastapi
bgi scan fastapi/ --out output/

# 3. Start the MCP context server (includes BGI-TWIN tools)
bgi mcp --graph output/bgi-graph.json --fuse-graph output/fuse-graph.json

# 4. Point your AI coding client at it (opencode.json in the repo dir):
# {
#   "mcp": {
#     "bgi": {
#       "command": "bgi",
#       "args": ["mcp", "--graph", "output/bgi-graph.json",
#                "--fuse-graph", "output/fuse-graph.json"]
#     }
#   }
# }

# 5. Run with a task-anchored prompt to trigger twin_context:
opencode  # → AI receives architecture summary + behavioral twins + seam + rubric

The three BGI-TWIN tools are available as MCP tools automatically once the server starts. No additional configuration needed.