InfoSeek - Deep Web Search & Archival

Overview

InfoSeek performs comprehensive web research on any subject (person, organization, product) across multiple search engines, deduplicates results, extracts clean content, and archives everything with full metadata in organized folders.

Prerequisites

Before executing a search task, verify these skills are installed:

import os
from pathlib import Path

workspace = os.environ.get('OPENCLAW_WORKSPACE')
skills_dir = Path(workspace) / 'skills'

required = ['baidu-search', 'tavily', 'Multi-Search-Engine', 'agent-browser-clawdbot-0.1.0']
missing = [s for s in required if not (skills_dir / s).exists()]

If any are missing, instruct the user to install them:

openclaw skills install baidu-search
openclaw skills install tavily-search
openclaw skills install multi-search-engine

Workflow

Phase 0: Task Setup

1. Confirm the search subject — name, organization, or product
2. Collect optional context — background info, time range, output format (default: .md), special requirements
3. Check dependencies — run the prerequisite check above
4. Create archive folder — run:

```bash

python scripts/infoseek_helper.py create-folder ""

```

Phase 1: Multi-Engine Deep Search

Execute searches across all available engines. Each engine runs independently.

1.1 Baidu Search (100+ pages)

Use the baidu-search skill:

• Query: " "
• Depth: 100+ pages
• Record: URL, title, website name, publish date for each result

1.2 Tavily Search

Use tavily_search tool:

query: "<subject> <background_context>"
search_depth: advanced
max_results: 50

1.3 Multi-Search-Engine

Use the multi-search-engine skill across multiple engines simultaneously.

1.4 Browser Deep-Crawl

For discovered URLs, use the browser tool to:

1. Open each page
2. Extract body content (filter ads, sidebars, comments)
3. Extract metadata: title, author, editor, date, website name

Phase 2: Deduplication

Run URL deduplication on all collected results:

python scripts/infoseek_helper.py deduplicate "<temp_results_file>"

The script normalizes URLs (remove www, tracking params, unify http/https, remove trailing slashes) and checks against the SQLite database to skip duplicates.

Phase 3: Content Extraction & Storage

For each unique URL:

1. Extract content using the browser tool — get title, body, metadata
2. Filter content — remove ads, sidebars, navigation, comments, related articles, footers
3. Generate filename:

```bash

python scripts/infoseek_helper.py generate-filename \

--date "" --title "