A precise, structured research pipeline — not a bulk dump, but a curated read-and-select
workflow that mirrors how an expert researcher actually works.
Step 1 · Search arXiv for the topic
Step 2 · Open 5 promising papers → PDF → Markdown via markitdown
Step 3 · Read title + abstract + intro of each
Step 4 · Claude selects the best 2 (explains why, drops the rest)
Step 5 · Search web: explanation · GitHub · survey · citations
Step 6 · Write a 5-bullet summary of what was learned
pip install requests markitdown
pip install curl_cffi beautifulsoup4 lxml fake-useragent
chmod +x scripts/owl.py
cp scripts/owl.py /usr/local/bin/owl
owl "diffusion models"
owl "LoRA fine-tuning" --category cs.LG --output report.md
owl "protein structure prediction" --since year
owl "quantum error correction" --arxiv-n 8
For Windows user:
python scripts/owl.py "AI agent RAG" --arxiv-n 8 --output report.md
| Flag | Default | Description |
|---|---|---|
| ------ | --------- | ------------- |
query | (required) | Research topic |
--arxiv-n N | 5 | arXiv candidates to fetch (top N are used as pool) |
--category CAT | all | arXiv category, e.g. cs.LG, q-bio.BM |
--sort | relevance | relevance, lastUpdatedDate, submittedDate |
--since RANGE | all time | Google recency: hour, day, week, month, year |
--papers-dir DIR | /tmp/owl_papers | PDF + markdown cache directory |
--output FILE | terminal only | Save Markdown summary to FILE |
--no-stream | off | Wait for full Claude response |
--serper-key KEY | env var | Override SERPER_API_KEY |
--anthropic-key KEY | env var | Override ANTHROPIC_API_KEY |
search_arxiv(query, n, category, sort_by) → list of paper dicts with
id, title, authors, abstract, published, categories, url, pdf_url.
Take top 5 candidates. For each: download PDF to --papers-dir, convert to
Markdown using markitdown (Python lib preferred, CLI fallback). Extract first
8 KB of text = roughly title + abstract + introduction. Falls back to abstract
only if PDF download or conversion fails.
Display truncated abstract+intro for each candidate. All 5 intro texts are
passed to Claude in Step 4.
Claude returns `{"selected": [i, j], "reason_1": "...", "reason_2": "...",
"dropped": "..."}`. The model used is printed in the step output and checklist.
Selected papers then get their full markdown (up to 40 KB) loaded for Step 6.
Runs 4 targeted Serper queries and fetches page content for the top 2 results
of each:
| Search type | Query pattern |
|---|---|
| ------------- | --------------- |
| explanation | {topic} explained |
| github | {topic} github implementation |
| survey | {topic} survey review paper |
| citations | {topic} highly cited papers results |
Builds a prompt with both full papers + all 4 web search result sets, then
calls LLM api. Output structure:
# 🦉 OWL Research Summary: {topic}
## Selected Papers
[P1] Title — Authors — URL — PDF
[P2] Title — Authors — URL — PDF
## 5-Bullet Summary
• **What it is**: ...
• **How it works**: ...
• **Key results**: ...
• **Tools & code**: ...
• **Open questions**: ...
## Web Sources
...
At the end of every run, owl prints a checklist confirming every step completed
successfully. Verify all 6 items are marked [✓] before trusting the output.
────────────────────────────────────────────────────────────────────────
✅ RESEARCH CHECKLIST
────────────────────────────────────────────────────────────────────────
[✓] Step 1 Search arXiv 5 papers found
[✓] Step 2 Open 5 papers via markitdown 5 PDFs converted
[✓] Step 3 Read title + abstract + intro 5 papers scanned
[✓] Step 4 Keep best 2 2 selected (2 with full text)
→ Paper title one…
→ Paper title two…
[✓] Step 5 Web search (4 queries) 16 results fetched
→ explanation Top result title…
→ github Top result title…
→ survey Top result title…
→ citations Top result title…
[✓] Step 6 5-bullet summary written by Claude
→ /path/to/report.md
────────────────────────────────────────────────────────────────────────
| Step | What to verify |
|---|---|
| ------ | --------------- |
| Step 1 | At least 1 paper found — if 0, the arXiv query returned nothing; try rephrasing or removing --category |
| Step 2 | PDFs converted — a ⚠ using abstract only warning here means markitdown failed for that paper; the run continues but that paper's selection is abstract-only |
| Step 3 | Count matches Step 2 — should always be 5 (or --arxiv-n if overridden) |
| Step 4 | Exactly 2 selected, model name visible — the model shown is whatever GET /v1/models returned as the best available; if it shows the fallback claude-sonnet-4-20250514 the models list API may have failed silently |
| Step 5 | 4 search types listed, each with a top result — a missing type means that Serper query failed silently |
| Step 6 | "written by Claude" confirms the API call completed — if the summary is truncated, raise max_tokens in the source |
⚠ — markitdown is not installed (pip install markitdown) or PDFs are blocked by arXiv rate limiting; wait and retryOPENROUTER_API_KEYSERPER_API_KEY--arxiv-n or use --no-stream--papers-dir — re-runs on the same paper are instant--arxiv-n can be raised to 8–10 for broader candidate pools before selection共 1 个版本