Lowyat Forum Research Tool
End-to-end research pipeline: Search → Scrape → Analyze
Workflow
Step 1: Understand the user's research topic
- Ask clarifying questions if needed (e.g. what specifically they want to learn)
- Break the topic into 3-5 search keyword variations
Step 2: Search for relevant threads
- Use
WebSearch with site:forum.lowyat.net to find threads - Use
allowed_domains: ["forum.lowyat.net"] to filter results - Run multiple searches in parallel with different keyword angles
- Present the most relevant threads to the user with titles and URLs
- Let the user pick which threads to scrape, or recommend the best ones
Step 3: Scrape the selected threads
- The scraper script (
datascraping.py) should be in the project root - Install Python dependencies:
pip install requests beautifulsoup4 html5lib openpyxl tqdm
Or if you have uv installed:
uv sync
- Run the scraper for each thread:
python datascraping.py <TOPIC_URL>
- IMPORTANT: Do NOT include
/all or /+N suffixes in the URL — just use the base topic URL (e.g. https://forum.lowyat.net/topic/5411252) - To scrape multiple threads, run them sequentially (one at a time) to be respectful to the server. Only run up to 3 in parallel if the user explicitly asks for speed, using
& and wait - Output:
.xlsx files with columns: Name, Date, Comment
Step 4: Analyze the scraped data
- Read the scraped
.xlsx files to understand the forum discussions - Synthesize findings across all threads into a structured summary
- Organize insights by the user's research questions
- Include: consensus opinions, brand recommendations, price ranges, warnings, and specific user experiences
- Cite which thread/user said what when relevant
Scraper Details
- Forum uses 20 posts per page, paginated via
/+N URL suffix - Scraper auto-detects total pages and crawls all of them
- Random 0.5–2s delay between page requests
- Saves incrementally after each page — safe to interrupt
- If
.xlsx already exists, it resumes by appending
Tips for good searches
- Use brand names:
site:forum.lowyat.net Toto toilet - Use Malay keywords too:
site:forum.lowyat.net kipas exhaust tandas - Add "recommendation" or "review":
site:forum.lowyat.net water heater recommendation - Search by location:
site:forum.lowyat.net bathroom shop KL Selangor - Try year filters for recency:
site:forum.lowyat.net smart toilet 2024 2025
Example usage
User: "I want to research mechanical keyboards on Lowyat"
- Search with variations:
mechanical keyboard recommendation, cherry mx switch, keychron Malaysia, custom keyboard - Present top threads to user
- Scrape selected threads in parallel
- Read the xlsx files and provide analysis: popular brands, price ranges, where to buy, common complaints
Links
Disclaimer
Scraped data contains publicly available usernames, dates, and comments from forum.lowyat.net. This tool is intended for personal research purposes only. Users are responsible for how they store, share, and use the scraped data in compliance with applicable privacy laws and Lowyat forum's terms of service.