LangSmith CLI Skill

CLI: scripts/langsmith.py. Requires LANGSMITH_API_KEY in env (or ~/.zshrc).

No second API key needed — the ask command fetches and formats traces as structured context for your agent to analyze. No trace data is sent to any third-party LLM.

Commands

Tier 0 — Ask (agent Q&A over traces)

python3 scripts/langsmith.py ask "<question>" --project <name> [--since 24h] [--limit 50]

Fetches recent runs and prints them as structured JSON context. Your agent reads the output and answers the question — no external LLM calls, no data leaving your machine beyond the LangSmith API.

Examples:

• ask "why is my chain slow this week" --project my-project
• ask "what do failing runs have in common" --project my-project --since 7d
• ask "did the system prompt change on Friday affect output quality" --project my-project

Tier 1 — Situational Awareness

python3 scripts/langsmith.py runs <project> [--since 2h] [--status error|success] [--limit 20]
python3 scripts/langsmith.py cost <project> [--since 7d]       # token spend by chain/node
python3 scripts/langsmith.py latency <project> [--since 24h]   # p50/p95/p99 per run name

Tier 2 — Before/After Comparisons

python3 scripts/langsmith.py diff <project> --before <ISO_date> --after <ISO_date>
python3 scripts/langsmith.py prompt-diff <run_id_a> <run_id_b>

diff compares avg latency, error rate, cost, output length across two time windows.

prompt-diff shows side-by-side system prompts + outputs for two specific runs.

Tier 3 — Deep Analysis (stubs, expand as needed)

python3 scripts/langsmith.py cluster-failures <project> [--since 7d]
python3 scripts/langsmith.py replay <run_id>

Auth Setup

export LANGSMITH_API_KEY=<your-key>
# or add to ~/.zshrc

Test with: python3 scripts/langsmith.py runs --limit 3

Security & Data Flow

This skill makes outbound network requests only to api.smith.langchain.com (the LangSmith API). That's it.

• LANGSMITH_API_KEY — sent as an HTTP header to api.smith.langchain.com only. Never logged or stored.
• Trace data — fetched from LangSmith and printed to stdout for your agent to read. No trace data is sent to any third-party LLM or external service.
• No second API key required — the ask command outputs structured trace context for your existing agent to analyze, rather than making its own LLM calls.
• No telemetry — the script collects no usage data.

The script is ~300 lines of pure Python with no obfuscation. Audit it at scripts/langsmith.py.

API Reference

See references/langsmith-api.md for endpoint details and run object schema.