GraphRAG Builder Skill

Turns any documentation website into a runnable MCP knowledge server in 5 pipeline steps,

each run on the user's local machine using scripts in the scripts/ folder.

Quick Reference

All scripts require Python 3.10+ and auto-install their own dependencies on first run.

Step 0: Clarify Requirements

Before running anything, ask the user:

• URL: Which site to crawl (required — starting page)
• Depth: How many link-hops to follow (default 3; suggest 2 for large sites)
• Model: Which Claude model for concept extraction — haiku (fast/cheap) or sonnet (higher quality). Default: haiku

Set the output slug from the URL: https://strudel.cc → strudel-cc-mcp.

Step 1: Crawl (M1)

Provide this command for the user to run locally:

python scripts/crawl.py \
  --url <URL> \
  --max-depth <DEPTH> \
  --output ./output

What to expect:

• Creates output//raw_content/*.json (one per page)
• Creates output//crawl.json (state tracking)
• Prints a summary: pages crawled, JS fallbacks used, failures

Common issues:

• JS-heavy single-page apps → many Playwright fallbacks (normal, just slower)
• Rate limiting → add --rate-limit 1.5 to slow down
• First run needs: pip install playwright && playwright install chromium

Step 2: Extract Concepts (M2)

The user must set ANTHROPIC_API_KEY first. Provide this command:

ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \
  --input ./output/<slug>-mcp \
  --model haiku

Dry-run first (no API cost):

python scripts/extract_concepts.py --input ./output/<slug>-mcp --dry-run

This validates chunking quality before spending API budget. Show them chunk counts and section names from dry-run output.

What to expect:

• Processes ~2–5 pages/minute on haiku
• Creates output/-mcp/extracted/*.json (one per page)
• Each file contains chunks with: concepts, tags, code examples, prerequisites, relationships
• Skips already-extracted pages (safe to re-run after interruption)

Common issues:

• Pages showing no_chunks → likely JS-rendered content not captured; acceptable for a minority of pages
• API rate limiting → script retries automatically with exponential backoff
• --max-pages 10 flag to test on a small sample first

Re-running after a partial run:

python scripts/extract_concepts.py --input ./output/<slug>-mcp --model haiku
# (automatically skips already-extracted pages)

Force re-extraction of everything:

python scripts/extract_concepts.py --input ./output/<slug>-mcp --force

Step 3: Build Graph (M3)

python scripts/build_graph.py --input ./output/<slug>-mcp

What to expect:

• Reads all non-dry-run extracted/*.json files
• Deduplicates concept names (case-insensitive, strips trailing ())
• Creates output/-mcp/graph.json
• Prints node/edge counts by type

Healthy output looks like:

Pages:     46
Chunks:    357
Concepts:  200+
Total edges: 1000+
  MENTIONS       600+
  REQUIRES       100+
  HAS_CHUNK      357
  LINKS_TO       80+
  RELATED        40+

If concepts = 0 and "Skipped N dry-run files" appears, M2 hasn't been run with a real API key yet.

Step 4: Build Embeddings (M4)

python scripts/build_embeddings.py --input ./output/<slug>-mcp

First run downloads all-MiniLM-L6-v2 (~80MB, cached after that).

Add --smoke-test to query both collections immediately after building:

python scripts/build_embeddings.py --input ./output/<slug>-mcp --smoke-test

What to expect:

• Creates output/-mcp/embeddings/ with 5 numpy files (no database needed)
• Two indexes: chunks (semantic search) and concepts (concept lookup)

Step 5: Generate MCP Server (M5)

python scripts/generate_mcp_server.py --input ./output/<slug>-mcp

Outputs:

• output/-mcp/server.py — the runnable MCP server
• output/-mcp/mcp_config.json — Claude Desktop config snippet

Install into Claude Desktop:

1. Open ~/Library/Application Support/Claude/claude_desktop_config.json
2. Merge the contents of mcp_config.json into the "mcpServers" key
3. Restart Claude Desktop
4. The server name (e.g., strudel-cc) appears in Claude's available tools

Test the server standalone:

python output/<slug>-mcp/server.py
# Should print "Loading ... knowledge graph... Ready: N nodes, M edges"

The 8 MCP Tools

Once installed, Claude can use these tools against the knowledge base:

Complete Pipeline Command Sequence

For a fresh install, provide the user with all commands in order:

# 0. Install system deps (once)
pip install requests beautifulsoup4 lxml playwright anthropic \
    networkx numpy sentence-transformers mcp
playwright install chromium

# 1. Crawl
python scripts/crawl.py --url <URL> --max-depth 3 --output ./output

# 2. Extract concepts (dry-run first)
python scripts/extract_concepts.py --input ./output/<slug>-mcp --dry-run
# Then real run:
ANTHROPIC_API_KEY=sk-ant-... python scripts/extract_concepts.py \
  --input ./output/<slug>-mcp --model haiku

# 3. Build graph
python scripts/build_graph.py --input ./output/<slug>-mcp

# 4. Build embeddings
python scripts/build_embeddings.py --input ./output/<slug>-mcp --smoke-test

# 5. Generate server
python scripts/generate_mcp_server.py --input ./output/<slug>-mcp

# 6. Test server
python output/<slug>-mcp/server.py

Output Directory Layout

output/<slug>-mcp/
├── crawl.json              State tracking (incremental re-runs)
├── raw_content/            One JSON per crawled page (HTML + links)
├── extracted/              One JSON per page (chunks + LLM concepts)
├── graph.json              networkx knowledge graph
├── embeddings/             numpy indexes (chunks.npy, concepts.npy + JSON)
├── server.py               The runnable MCP server ← share this
└── mcp_config.json         Claude Desktop config snippet ← install this

The entire output/-mcp/ folder is the deliverable. The user can move it anywhere

as long as server.py, graph.json, and embeddings/ stay together.

Troubleshooting

"No module named X" → The script auto-installs deps, but if it fails:

pip install <package> --break-system-packages

Crawl gets 0 pages → Check robots.txt and try --force to bypass the crawl cache.

extract_concepts produces tiny concepts count → The page content may be JS-only.

Check fetched_with field in raw_content/*.json — pages fetched via requests with

very little text should have been picked up by Playwright. Re-crawl with --force.

Server fails to start → Run python output/-mcp/server.py directly and check

stderr for import errors. Most common cause: mcp package not installed.

Claude Desktop doesn't show the server → Verify the path in mcp_config.json is

absolute and the file exists. Restart Claude Desktop after any config change.

Deferred Features

See TODO.md for planned improvements including:

• YouTube transcript fetching
• Neo4j export for large graphs
• OpenAI/Voyage embedding API support
• Scheduled re-crawls
• Graph visualization