Skroller - Social Media Content collection

Automate the collection and analysis of publicly available social media content. This skill handles content collection, filtering, and export with compliance safeguards.

⚖️ Legal & Compliance Requirements

Before using this skill:

1. Review Platform ToS - Each platform has different rules about automated access
2. Check robots.txt - Respect disallowed paths
3. Rate Limiting - Stay within platform rate limits to avoid service disruption
4. Privacy Laws - GDPR, CCPA, and other regulations apply to personal data
5. Permitted Use - Research, personal analysis, competitive intelligence (where allowed)
6. Prohibited Use - Spam, harassment, data resale, manipulation, bypassing auth

Data Protection:

• Anonymize personal data when storing
• Honor deletion requests (GDPR Art. 17)
• Limit retention to necessary periods
• Document lawful basis for processing
• Do not scrape sensitive personal data

Core Capabilities

• Content collection - Gather publicly available posts from platforms
• Data extraction - Pull text, timestamps, engagement metrics, author info
• Smart filtering - Filter by keywords, hashtags, date ranges, engagement thresholds
• Deduplication - Track seen posts to avoid duplicates across sessions
• Export formats - JSON, CSV, Markdown, or direct to note apps (Bear/Obsidian)
• Rate limiting - Respect platform constraints and avoid service disruption

Platform Support

Quick Start

Basic Scroll and Extract

# Scroll Twitter feed, extract 50 posts about "AI"
node scripts/skroller.js --platform twitter --query "AI" --limit 50 --output posts.json

Monitor Brand Mentions

# Monitor Reddit for brand mentions, export to Markdown
node scripts/skroller.js --platform reddit --query "mybrand" --format markdown --output mentions.md

Competitive Research

# Scroll competitor Instagram, capture top posts by engagement
node scripts/skroller.js --platform instagram --profile @competitor --min-likes 1000 --output competitor-posts.json

Scripts

scripts/skroller.js - Main scrolling engine

JavaScript script using Playwright for browser automation.

# Using npm scripts (recommended)
npm run scroll -- --platform twitter --query "AI" --limit 50 --output posts.json

# Direct execution
node scripts/skroller.js --platform twitter --query "AI" --limit 50 --output posts.json

Options:

• --platform - Target platform (required): twitter, reddit, instagram, tiktok, linkedin, youtube, producthunt, medium, github, pinterest
• --query - Search keyword/hashtag
• --profile - Specific profile to scroll
• --limit - Max posts to scrape (default: 50)
• --min-likes - Filter by minimum engagement
• --format - Output format: json, csv, markdown (default: json)
• --output - Output file path
• --screenshot - Capture screenshots for debugging
• --dedupe - Skip previously seen posts

scripts/feed-digest.js - Generate digests

Creates summary digests from exported post data.

npm run digest -- --input posts.json --output digest.md
# or: node scripts/feed-digest.js --input posts.json --output digest.md

scripts/export-to-notes.js - Unified note app exporter

Exports scraped posts to multiple note applications with a single command.

Supported apps: Bear, Obsidian, Notion, Apple Notes, Evernote, OneNote, Google Keep, Roam Research, Logseq, Anytype

# Using npm scripts (recommended)
npm run export -- --input posts.json --app obsidian --vault ~/Documents/Obsidian

# Direct execution
node scripts/export-to-notes.js --input posts.json --app bear --tags "ai,research"
node scripts/export-to-notes.js --input posts.json --app notion --api-key $NOTION_TOKEN
node scripts/export-to-notes.js --input posts.json --app apple-notes
node scripts/export-to-notes.js --input posts.json --app evernote --output export.enex
node scripts/export-to-notes.js --input posts.json --app one-note --access-token $MS_TOKEN
node scripts/export-to-notes.js --input posts.json --app keep --output keep.html
node scripts/export-to-notes.js --input posts.json --app roam --output roam.md
node scripts/export-to-notes.js --input posts.json --app logseq --vault ~/Documents/Logseq
node scripts/export-to-notes.js --input posts.json --app anytype --output anytype.md
node scripts/export-to-notes.js --input posts.json --app obsidian --dry-run

Configuration: Set defaults in .skroller-config.json:

{
  "export": {
    "defaultApp": "obsidian",
    "vault": "~/Documents/Obsidian",
    "folder": "Skroller",
    "notionDatabaseId": "<db-id>"
  }
}

Requirements by app:

• Bear: grizzly CLI (go install github.com/tylerwince/grizzly/cmd/grizzly@latest)
• Obsidian: Vault path
• Notion: API key (create at notion.so)
• Apple Notes: macOS with Notes app
• Evernote: Manual ENEX import
• OneNote: Microsoft Graph access token
• Google Keep: Manual HTML import
• Roam Research: Markdown import (drag MDL file into Roam)
• Logseq: Vault path (writes to pages/ folder)
• Anytype: Markdown import (use Anytype Import feature)

Configuration

.skroller-config.json

Store default settings:

{
  "defaultLimit": 50,
  "scrollDelayMs": 1500,
  "userAgent": "Mozilla/5.0 ...",
  "platforms": {
    "twitter": {
      "loginRequired": false,
      "rateLimit": "100 requests/hour"
    },
    "instagram": {
      "loginRequired": true,
      "rateLimit": "50 requests/hour"
    }
  },
  "export": {
    "defaultFormat": "json",
    "includeImages": true,
    "includeMetrics": true
  }
}

Authentication

Some platforms require login. Store credentials securely:

# For platforms requiring auth, set environment variables
export SKROLLR_TWITTER_COOKIE="<auth cookie>"
export SKROLLR_INSTAGRAM_USER="<username>"
export SKROLLR_INSTAGRAM_PASS="<password>"

Security note: Never commit auth files. Use .env with .gitignore.

Output Structure

JSON Output (default)

{
  "platform": "twitter",
  "query": "AI",
  "scrapedAt": "2026-03-14T20:30:00Z",
  "posts": [
    {
      "id": "1234567890",
      "author": "@username",
      "text": "Post content here...",
      "timestamp": "2026-03-14T18:00:00Z",
      "likes": 150,
      "retweets": 42,
      "replies": 12,
      "url": "https://twitter.com/...",
      "media": ["image1.jpg"],
      "hashtags": ["#AI", "#ML"]
    }
  ]
}

Markdown Output

## Twitter Posts: "AI" (2026-03-14)

### @username - 150 likes
Post content here...

[View post](https://twitter.com/...)

---

Filtering Strategies

Keyword Matching

• Exact match: "exact phrase"
• Boolean: AI AND (startup OR venture)
• Exclude: AI -crypto

Engagement Thresholds

• Filter low-quality: --min-likes 100
• Viral content: --min-shares 500

Time Windows

• Recent only: --date-from 2026-03-14
• Historical: --date-from 2026-01-01 --date-to 2026-01-31

Best Practices

Rate Limiting

• Add delays between scrolls: --delay 2000
• Respect platform limits (see config)
• Use proxies for high-volume scraping

Content Quality

• Filter by engagement to find signal
• Dedupe across sessions
• Export with timestamps for freshness tracking

Ethics & Compliance

• Check platform ToS before scraping
• Don't scrape personal data at scale
• Use for research/curation, not spam

Troubleshooting

Scroll stops early

• Increase --limit or check for login requirements
• Some platforms detect automation; add random delays

Missing content

• Some platforms lazy-load; increase scroll delay
• Try --screenshot to debug what's visible

Rate limited

• Reduce frequency; use --delay
• Check platform-specific rate limits in config

Integration with Other Skills

• bear-notes: Export via export-to-notes.js --app bear
• obsidian: Export via export-to-notes.js --app obsidian
• notion: Export via export-to-notes.js --app notion
• github: Create issues from scraped feedback/mentions

# Scroll and export to Obsidian in one command
node scripts/skroller.js --platform twitter --query "AI" --limit 20 --output ai.json && \
  node scripts/export-to-notes.js --input ai.json --app obsidian --vault ~/Documents/Obsidian

# Scroll and export to Notion
node scripts/skroller.js --platform reddit --query "startups" --limit 30 --output startups.json && \
  node scripts/export-to-notes.js --input startups.json --app notion --api-key $NOTION_TOKEN

# Use npm scripts for cleaner commands
npm run scroll -- --platform twitter --query "tech" --limit 25 --output tech.json
npm run export -- --input tech.json --app bear --tags "tech,research"

See Also

• references/platform-details.md - Platform-specific selectors and quirks
• references/rate-limits.md - Rate limit guidelines per platform
• assets/selector-cheatsheet.md - CSS selectors for each platform