Context Overflow Recovery v2.7

Target: Complex multi-step projects only. NOT for everyday tasks.

What This Skill Is

This skill implements a Context Engineering pipeline -- not just note-taking, but a structured 5-stage process for extending the AI's effective memory beyond its context window:

> Identify which tasks need tracking (>=5 steps, multi-system, or cross-session)

> Classify every piece of information as a Static Anchor (permanent, verbatim) or Dynamic Flow (compressible)

> Compress dynamic flow via progressive step collapsing, never touching static anchors

> Retrieve exact state on recovery via dual-tier file reads

> Persist after every substantive step with size-controlled snapshots

Like OS virtual memory extends RAM with disk, this skill extends the context window with persistent files. A context overflow is a "page fault"; loading progress.md is the "disk read". The illusion is seamless memory; the reality is strategic compression and retrieval.

Theoretical Foundation: Information Energy

The collapse priority system is grounded in information theory -- specifically the Entropy Gate framework from Stanford NLP research, which models each token as carrying a unit of "information energy." Compression is the selective removal of low-energy tokens while preserving high-energy ones, optimizing the trade-off between rate (how much we keep) and distortion (what we lose).

Our 6-level collapse priority is essentially an information energy ranking:

• Highest energy (never collapse): Key decisions, static configuration, project structure -- the "ground truth" tokens that define the entire project
• Medium energy (collapse with care): Completed steps, discussion conclusions -- valuable but reconstructable from artifacts
• Lowest energy (collapse first): Debug traces, error logs, intermediate outputs -- ephemeral by nature, regenerated when needed

This is NOT just a heuristic. It reflects the principle that information loss is not uniform -- losing a project's API endpoint is catastrophic, while losing a debug log is inconsequential.

Also: even models with 1M+ token context windows suffer from "Lost in the Middle" (Databricks, 2024) -- recall drops for information in the middle of long contexts, regardless of window size. This skill addresses a model architecture limitation, not just a window size problem.

The One Problem This Solves

Task 1 (complex project) ---> context overflows ---> session dies
                                                    |
                                                    v
Task 2 (same workspace)     ---> reads Task 1's disk snapshot
                              ---> continues from exact step where Task 1 stopped

If your task finishes in under ~10 conversation turns, you do NOT need this skill.

When to Use (Trigger Threshold)

USE this skill (any ONE of these is enough):

DO NOT use this skill (handle normally):

For non-tracked tasks: normal .workbuddy/memory/ daily log + workspace file history is sufficient. Completed tasks remain visible via WorkBuddy's task system and file timestamps.

Two-Tier File Architecture

<workspace_root>/
  progress.md                    # TIER 1: Workspace Overview (summary only)
  .progress/                     # TIER 2: Per-Task Detail
    task-{id}.md                # Active/completed task snapshots
    archive/                    # Auto-archived old completed tasks

Tier 1: progress.md (Workspace Root, < 4KB)

Quick summary table of all tracked tasks + workspace conventions.

Tier 2: .progress/task-{id}.md (< 6KB per file)

Granular step-by-step progress for ONE complex task.

Boundary With Other Memory Systems

This skill has a specific scope. It does NOT replace other systems:

Rule of thumb: If it answers "where exactly did I stop?" → progress-tracker. If it answers "what did I learn?" → MEMORY.md. If it answers "what did I do today?" → daily log.

Core Protocol

Rule 1: Activation Check

Before creating ANY progress files, evaluate:

def should_track():
    # Auto-evaluate before initializing
    estimated_steps = estimate_complexity()
    if estimated_steps >= 5:
        return True
    if touches_multiple_systems():
        return True
    if user_explicitly_requested():
        return True
    return False  # Skip this skill entirely for simple work

If should_track() returns False → do not create any progress files. Proceed with normal workflow.

Rule 2: Dual-Write After Substantive Steps

After completing a substantive step in a tracked task:

1. Update Tier 2 (task detail file) — overwrite with latest state
2. Update Tier 1 (workspace overview) — bump timestamp + status line
3. Execute via Python script immediately, no user confirmation needed
4. Output brief confirmation only

Rule 3: Cross-Task Recovery

When starting a NEW session/task in a workspace that has existing progress files:

1. Read progress.md → get task registry
2. Read relevant .progress/task-{id}.md files → recover detailed state
3. Report to user: [RECOVERED] Found N tracked task(s):
• Task A: last at Step X (status)
• Task B: last at Step Y (status)
4. Continue seamlessly from recovered state

Rule 4: Size Control & Collapse Rules

• Tier 1 hard cap: 4KB. If exceeded → trim Global Pending Items and Notes sections first
• Tier 2 hard cap: 6KB per task file. If exceeded, apply these collapse rules in order:

Collapse priority (highest loss tolerance first):

1. Debug logs & error traces → keep only error type + resolution, discard stack traces and intermediate outputs
2. Discussion transcripts → keep only final conclusion, discard back-and-forth reasoning
3. Older completed steps (beyond last 5) → collapse into ONE summary line: [x] Steps 1-N: [one-line summary] ({date})
4. Recent completed steps (last 5) → keep full detail with timestamps -- these are the "warm cache"
5. Current step + Next steps → NEVER collapse -- this is the active execution state
6. Key Decisions + Static Anchors → NEVER collapse -- these are the immutable foundation

Code-specific compression rules:

• NEVER convert code into natural-language summaries (e.g., "the function validates input and returns an error" is useless for recovery)
• Preserve function/method signatures + parameter types verbatim
• Keep key logic skeleton: control flow structure, error handling branches, return values
• Use file path + line number references instead of copying full code blocks (e.g., auth.py:L42-58: JWT validation with refresh)
• When compressing code changes across steps, record only the diff intent: "added rate limiting to /api/search" rather than pasting the full diff
• This aligns with industry consensus (Tencent Cloud, LangChain): AST-level structure preservation > natural-language paraphrasing for code

What to NEVER write into either tier:

• Raw conversation transcripts (user messages / AI responses verbatim)
• Stack traces beyond the first identifying line
• Full file contents (use file paths + line references instead)

Rule 5: Auto-Archive

When a task status becomes completed:

• Keep its file in .progress/ for 7 days
• After 7 days: move to .progress/archive/ automatically
• If more than 5 completed+archived tasks exist: delete oldest archived task (keep most recent 5)
• This keeps .progress/ directory clean: only active + recently-completed tasks visible

Rule 6: Encoding & Safety

• UTF-8 always
• No emoji/non-ASCII in code comments or file content
• ASCII markers: [OK], [FAIL], [WARN], [RECOVERED], [SKIPPED], [ARCHIVED]
• Filenames: lowercase alphanumeric + hyphens only: task-wecom-bot-v2.md

File Content Templates

Tier 1: progress.md

# {Workspace} Progress Overview

**Workspace:** {path}
**Last Updated:** {datetime}
**Tracked Tasks:** N active | M completed

---

## Tracked Task Registry

| ID | Name | Status | Last Step | Detail File |
|----|------|--------|-----------|-------------|
| {id} | {name} | {status} | {step} | .progress/task-{id}.md |

## Global Pending Items
(cross-task items only)

## Conventions
(shared resource paths, naming rules, etc.)

Tier 2: .progress/task-{id}.md

# Task: {Name}

**ID:** {id} | **Status:** {status} | **Updated:** {datetime}

---

## Static Anchors (NEVER collapse or summarize these)
- (project structure, API endpoints, core config, key data models)
- (these remain verbatim for the entire task lifecycle)

## Completed Steps
- [x] Step N: Description ({datetime})
(keep last 5 full; collapse older into summary line)

## Current Step
- [ ] Step N+1: Description
  - Sub-step / blocker details

## Current Blockers / Open Questions
- [ ] (what is currently stuck or unknown)
- (critical for recovery -- the next session knows exactly where to resume thinking)

## Key Decisions (accumulated)
1. Decision A: rationale
2. Decision B: rationale

## Next Steps
1. ...
2. ...

## Artifacts
(files created/modified by this task)

## Recovery Metadata
- Last checkpoint: {datetime}
- Steps collapsed: none / "1-3 (detail lost, summary retained)"
- Information gaps: none / describe any known gaps

---
*Auto-generated by progress-tracker v2.7.*

Static Anchor Identification Checklist

Use this checklist to identify what qualifies as a Static Anchor for your project:

Quick test: Ask yourself -- "If the next session loses this piece of information, would it have to guess, or could it recover from the codebase?" If guessing is required, it is a Static Anchor. If recoverable by reading code or config files, it is Dynamic Flow.

Real-World Example (Tier2 Filled)

A real Tier2 file from an actual project, showing what a populated snapshot looks like:

# Task: WeCom Bot Integration

**ID:** wecom-bot | **Status:** in_progress | **Updated:** 2026-06-03 12:21

---

## Static Anchors (NEVER collapse or summarize these)
- App Token: Pfqvb1pWRawE6HsCbN6cyJOZncK
- Table ID: tblLj6IzdwDBAttB
- Export template: 走访记录_导出模板.xlsx (30 columns, A区19+B区11)
- Field alias map: 19 A-fields map 1:1; B-fields need FIELD_ALIAS_MAP at write time
- Model split: glm-5v-turbo (analysis) + glm-4-flash (KB Q&A)

## Completed Steps
- [x] Step 1: Set up project structure, create main.py + requirements.txt (2026-06-02 09:15)
- [x] Step 2: Implement WebSocket connection with auto-reconnect (2026-06-02 10:30)
- [x] Step 3: Build KB search module with two-stage strategy (2026-06-02 14:20)
- [x] Step 4: Integrate GLM-5V for visit analysis (Mode A/B/E) (2026-06-03 09:45)
- [x] Step 5: Fix field alias mapping bug + date parsing bug (2026-06-03 11:00)
[x] Steps 1-3: Project init, WS connection, KB search module (2026-06-02)

## Current Step
- [ ] Step 6: PyInstaller packaging into single exe (~38MB target)
  - Sub-step: spec file datas array for templates + .env bundling
  - Key decision: use sys._MEIPASS for resource path resolution at runtime

## Current Blockers / Open Questions
- [ ] Not yet tested: does .env path resolution work inside PyInstaller bundle?
- [ ] Need to verify:飞书API rate limit for batch write operations

## Key Decisions (accumulated)
1. Switched from Webhook+cpolar to WebSocket long-connection (cpolar URL changes on restart)
2. DuckDuckGo failed on Python 3.13+Windows; Bing had poor Chinese results -> chose Baidu
3. Mode A (image parsing) skipped due to im:image permission -> Mode E covers the use case
4. Date parse failure returns None instead of 0 to avoid 1970/01/01 dirty data

## Next Steps
1. Package exe and copy to Desktop
2. E2E test: send bitable link -> AI analyzes -> writes backend -> exports Excel
3. Generate usage documentation (Word format)

## Artifacts
- kb_web_service.py (main bot, ~1200 lines)
- visit_parser.py (analysis module, ~580 lines)
- KB_Bot_v4_3.exe (38MB, Desktop)
-走访记录_导出模板.xlsx (export template, 30 columns)

## Recovery Metadata
- Last checkpoint: 2026-06-03 12:21
- Steps collapsed: 1-3 (detail lost, summary retained)
- Information gaps: none identified

---
*Auto-generated by progress-tracker v2.7.*

Notice how:

• Static Anchors preserve API tokens, table IDs, and config that must never be summarized away
• Current Blockers tell the next session exactly what to investigate first
• Older steps (1-3) are collapsed into one summary line
• Recent steps (4-5) keep full detail with timestamps
• Recovery Metadata is transparent about what was lost during compression
• Total file size stays well under 6KB

Common Anti-Patterns (Mistakes to Avoid)

Anti-Pattern 1: Tracking Everything

Mistake: Activating progress tracking for every task, even simple ones.

Consequence: progress.md bloats with noise; simple tasks drown out the important ones; recovery becomes slower and less useful.

Correct: Strictly follow the >=5-step threshold. Simple tasks belong in daily logs only.

Anti-Pattern 2: Never Cleaning Up Completed Tasks

Mistake: Letting .progress/ accumulate dozens of old task files.

Consequence: Recovery scans take longer; directory becomes cluttered; hard to spot which tasks are actually active.

Correct: Rely on the auto-archive mechanism. Periodically review .progress/archive/ and confirm the 7-day purge is working.

Anti-Pattern 3: Writing Conversation Transcripts

Mistake: Copying user messages and AI responses verbatim into task files.

Consequence: Tier2 file hits the 6KB cap almost immediately; key decisions get buried in chat noise.

Correct: Extract only decisions, outcomes, and next steps. Raw conversation belongs in memory/YYYY-MM-DD.md if it needs preserving.

Anti-Pattern 4: Parallel Sessions on Same Workspace

Mistake: Running two AI sessions in the same workspace, both writing to progress files.

Consequence: Write conflicts; one session overwrites the other's updates; state becomes inconsistent or corrupted.

Correct: One workspace = one active session at a time. If you need parallel work, use separate workspaces or pause tracking in one session.

Anti-Pattern 5: Tier1 Becomes a Second Tier2

Mistake: Putting detailed step information into progress.md (Tier1) instead of just the summary table.

Consequence: Tier1 exceeds 4KB cap quickly; it no longer serves as a quick overview.

Correct: Tier1 = registry table + conventions only. All detail goes into Tier2.

Troubleshooting Guide

Error Recovery Procedures

When things go wrong, follow these step-by-step recovery procedures. They are ordered by severity.

Recovery 1: progress.md is corrupted or empty

Severity: Medium. Tier1 is recoverable from Tier2.

Steps:

1. Do NOT delete .progress/ -- the Tier2 files are your lifeline
2. List all Tier2 files: ls .progress/task-*.md
3. For each file, read the first 5 lines to get task name and status
4. Rebuild Tier1 by creating a fresh progress.md with the registry table populated from Tier2 headers
5. Verify: the rebuilt Tier1 should have one row per Tier2 file

Recovery 2: A specific Tier2 file is corrupted

Severity: Low. One task lost, others unaffected.

Steps:

1. Check progress.md registry -- it has the task name and last known step
2. If the task has code artifacts, reconstruct from git history: git log --oneline --
3. If no artifacts, recreate Tier2 from Tier1's registry row + Recovery Metadata
4. Mark Recovery Metadata as information_gaps: [recovered from Tier1 summary, detail may be incomplete]

Recovery 3: .progress/ directory was accidentally deleted

Severity: High. All task detail is lost.

Steps:

1. Check if progress.md (Tier1) still exists -- it has the summary registry
2. If Tier1 exists: recreate .progress/archive/ directory, then reconstruct each Tier2 from Tier1 rows
3. If both are lost: check memory/YYYY-MM-DD.md for daily logs -- they may contain task progress notes
4. As last resort: check git history for any committed progress files

Recovery 4: File write fails (permission denied / disk full)

Severity: Low. Usually recoverable.

Steps:

1. Check disk space: df -h (Linux/Mac) or check drive properties (Windows)
2. Check file permissions: ls -la progress.md .progress/
3. If another process holds a lock, close it and retry
4. If workspace is on a network drive, try a local directory as fallback
5. After fixing: run the Self-Check Script to verify system health

Recovery 5: Tier1 or Tier2 exceeds size limit

Severity: Low. Requires manual compression.

Steps:

1. Identify the oversized file with the Self-Check Script
2. Read the file and identify which sections are consuming the most space
3. Apply collapse rules:
• Completed steps older than 3 rounds: collapse to single-line summaries
• Debug logs and error traces: remove entirely (regenerable)
• Static Anchors: NEVER touch these
4. If Tier1 is oversized, check if detail leaked in -- move detail to Tier2

Recovery 6: Information loss detected after collapse

Severity: Medium. Some data is missing but recoverable.

Steps:

1. Read Recovery Metadata -- it lists what was collapsed and what gaps exist
2. Cross-reference with memory/YYYY-MM-DD.md daily logs
3. If the lost info is about code: read the actual source files directly
4. If the lost info is a decision: check git commit messages or conversation history
5. After recovery: update Recovery Metadata to mark the gap as resolved

Self-Check Script

Run this in the workspace root to verify the progress system is healthy:

# Check Tier1 exists and is within size limit
if [ -f "progress.md" ]; then
    SIZE=$(wc -c < progress.md)
    if [ "$SIZE" -gt 4096 ]; then
        echo "[WARN] progress.md is ${SIZE} bytes (limit: 4096)"
    else
        echo "[OK] progress.md: ${SIZE} bytes"
    fi
else
    echo "[INFO] No progress.md found (normal if tracking not active)"
fi

# Check Tier2 directory and file sizes
if [ -d ".progress" ]; then
    ACTIVE=$(find .progress -maxdepth 1 -name "task-*.md" | wc -l)
    ARCHIVED=$(find .progress/archive -name "task-*.md" 2>/dev/null | wc -l)
    echo "[OK] Active tasks: ${ACTIVE}, Archived: ${ARCHIVED}"
    find .progress -name "task-*.md" -size +6k 2>/dev/null | while read f; do
        echo "[WARN] Oversized: $f"
    done
    # Check for archive directory
    [ -d ".progress/archive" ] || echo "[WARN] .progress/archive/ directory missing"
else
    echo "[INFO] No .progress/ directory (normal if tracking not active)"
fi

Windows (PowerShell) equivalent:

if (Test-Path "progress.md") {
    $size = (Get-Item "progress.md").Length
    if ($size -gt 4096) { Write-Host "[WARN] progress.md is $size bytes (limit: 4096)" }
    else { Write-Host "[OK] progress.md: $size bytes" }
} else { Write-Host "[INFO] No progress.md found" }

if (Test-Path ".progress") {
    $active = (Get-ChildItem ".progress" -Filter "task-*.md" -File).Count
    $archived = (Get-ChildItem ".progress\archive" -Filter "task-*.md" -File -ErrorAction SilentlyContinue).Count
    Write-Host "[OK] Active tasks: $active, Archived: $archived"
    Get-ChildItem ".progress" -Filter "task-*.md" | Where-Object { $_.Length -gt 6144 } | ForEach-Object {
        Write-Host "[WARN] Oversized: $($_.Name)"
    }
} else { Write-Host "[INFO] No .progress/ directory" }

Quick Start (30 Seconds)

Prerequisites: A workspace directory you can write files to. That's it.

Step 1: Initialize (copy & paste)

Create the directory structure and Tier1 overview file:

mkdir -p .progress/archive

Create progress.md at workspace root with this content:

# Project Progress Overview

**Workspace:** (your project name)
**Last Updated:** (auto-fill)
**Tracked Tasks:** 0 active | 0 completed

---

## Tracked Task Registry

| ID | Name | Status | Last Step | Detail File |
|----|------|--------|-----------|-------------|

## Global Pending Items
(none yet)

## Conventions
- Tier1 cap: 4KB | Tier2 cap: 6KB per task
- Auto-archive: completed tasks after 7 days
- Encoding: UTF-8, ASCII markers only

Step 2: Create your first task

Create .progress/task-001.md with this content:

# Task: (Your Task Name)

**ID:** 001 | **Status:** pending | **Updated:** (auto-fill)

---

## Static Anchors (NEVER collapse or summarize these)
(Add project structure, API endpoints, core config here as you discover them)

## Completed Steps
(none yet)

## Current Step
- [ ] Step 1: (First step description)

## Current Blockers / Open Questions
(none yet)

## Key Decisions
(none yet)

## Next Steps
1. ...

## Artifacts
(none yet)

## Recovery Metadata
- Last checkpoint: (auto-fill)
- Steps collapsed: none
- Information gaps: none

---
*Auto-generated by progress-tracker v2.7.*

Step 3: Update after each step

After completing any step:

1. Move it from "Current Step" to "Completed Steps" (collapse older steps if needed)
2. Update progress.md registry table row
3. If stuck, add to "Current Blockers"

Done! Your progress system is running.

Tips:

• You don't need to create task files manually -- the AI agent does this automatically when it detects a complex project
• Run the Self-Check Script periodically to verify file health
• If something breaks, follow the Error Recovery Procedures above

FAQ

Q: Can I use this with any AI coding assistant?

A: Yes. This skill works with any tool that can read/write files -- Claude Code, Cursor, Windsurf, Copilot, WorkBuddy, or plain terminal. No platform-specific features required.

Q: What happens if context overflows mid-write?

A: Worst case: one Tier file gets a truncated write. Since Tier1 and Tier2 are independent, the other file is always intact. Rebuild the corrupted file from its sibling.

Q: Should I track tasks in nested subdirectories?

A: No. All progress files live at the workspace root. This makes them easy to find and prevents path confusion during recovery.

Q: Can I rename or reorganize task IDs?

A: Yes, but update the Tier1 registry to match. The system only relies on filename matching -- there is no database or index.

Q: How is this different from just using git commits?

A: Git tracks code changes. This tracks task execution state (which step am I on, what's blocked, what decisions were made). They complement each other.

Progress Tracker v2.7 by LLLCAF -- Track what matters, skip what doesn't.

Session Recovery Template

Runs automatically when skill loads in any new session:

import os, glob

prog_file = 'progress.md'
prog_dir = '.progress'

if os.path.exists(prog_file):
    print('[RECOVERED] Workspace has tracked progress:')
    # read and display progress.md summary
else:
    # No tracked tasks exist in this workspace
    # This is normal for simple-task workspaces
    print('[INFO] No tracked tasks. Working in standard mode.')
    return  # Skip all progress operations

if os.path.exists(prog_dir):
    active = glob.glob(os.path.join(prog_dir, 'task-*.md'))
    archived = glob.glob(os.path.join(prog_dir, 'archive', 'task-*.md'))
    print('[RECOVERED] %d active + %d archived task(s)' % (len(active), len(archived)))
    for tf in sorted(active):
        print('  ACTIVE: %s (%d bytes)' % (os.path.basename(tf), os.path.getsize(tf)))

Recovery protocol (beyond auto-scan):

1. For each active task, read the Recovery Metadata section first -- it tells you what was collapsed and what gaps exist
2. Static Anchors are your ground truth -- trust them verbatim, never try to reconstruct from memory
3. Current Blockers are your starting point -- address these first before continuing normal steps

Workflow Examples

Scenario A: Simple Task (NO tracking)

User: "Rename this file to report-final.pdf"
AI: Does it in 1 turn. Done.
     No progress files created.
     Normal memory/daily-log handles it if needed.

Scenario B: Complex Project With Overflow (FULL tracking)

Task 1: Building WeCom bot (20+ turns, 12 steps done)
        -> hits context limit -> dies

Task 2: New session, same workspace
AI: Loads skill -> finds progress.md
   -> reads task-wecom-bot-v1.md (1916 bytes)
   -> reports: "Recovered: WeCom bot at Step 13, 6 decisions preserved"
   -> continues from Step 13

Scenario C: Mixed Workspace (some tracked, some not)

Same workspace, different tasks:
- Task A: "Fix typo in doc" -> NOT tracked (simple)
- Task B: "Deploy bot to production" -> TRACKED (6 steps, multi-system)
- Task C: "Answer question about API" -> NOT tracked (Q&A)

progress.md shows only Task B. Tasks A and C are invisible to
the tracker but fully accessible via file history and daily logs.

AI Agent Checklist

When this skill is loaded:

1. [ ] Evaluate: does current work meet trigger threshold? If NO → skip entirely, proceed normally
2. [ ] If YES: check if progress.md exists → read it
3. [ ] Check .progress/ for relevant task files
4. [ ] Identify/create current task entry
5. [ ] Read Recovery Metadata first → understand what was collapsed and what gaps exist
6. [ ] Read Static Anchors → these are your ground truth, never question them
7. [ ] Read Current Blockers → address these first before normal steps
8. [ ] Report recovered state to user
9. [ ] After each substantive step: dual-write (if still in threshold)
10. [ ] Update Recovery Metadata (checkpoint time, collapsed steps, gaps)
11. [ ] On completion: mark status = completed (auto-archive kicks in after 7 days)
12. [ ] Respect boundary: don't duplicate what belongs in MEMORY.md or daily log

Progress Tracker v2.6 — Track what matters, skip what doesn't.