Sci Data Extractor

You are a professional scientific literature data extraction assistant, helping users extract structured data from scientific paper PDFs.

Core Features

PDF Content Extraction

• Extract text from PDFs using Mathpix OCR or PyMuPDF
• Support for formula and table recognition

Data Extraction

• Use LLMs (Claude/GPT-4o/compatible APIs) to extract structured data from literature
• Automatically identify field types and data structures
• Support custom extraction rules and prompts

Output Formats

• Markdown tables
• CSV files

Installation

Prerequisites

• Python 3.8+
• pip package manager

Setup Steps

1. Install Python dependencies (choose one method):

Method 1: Using uv (Recommended - Fastest)

```bash

# Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

# Create virtual environment and install dependencies

cd /path/to/sci-data-extractor

uv venv

source .venv/bin/activate # Linux/macOS

# or .venv\Scripts\activate # Windows

uv pip install -r requirements.txt

```

Method 2: Using conda (Best for scientific/research users)

```bash

cd /path/to/sci-data-extractor

conda create -n sci-data-extractor python=3.11 -y

conda activate sci-data-extractor

pip install -r requirements.txt

```

Method 3: Using pip directly (Built-in, no extra installation)

```bash

cd /path/to/sci-data-extractor

pip install -r requirements.txt

```

1. Configure API credentials:

```bash

# Copy example configuration

cp .env.example .env

# Edit .env and add your API key

# Get API key from: https://console.anthropic.com/

EXTRACTOR_API_KEY=your-api-key-here

EXTRACTOR_BASE_URL=https://api.anthropic.com

EXTRACTOR_MODEL=claude-sonnet-4-5-20250929

EXTRACTOR_MAX_TOKENS=16384

```

1. Optional: Configure Mathpix OCR (for high-precision OCR):

```bash

# Get credentials from: https://api.mathpix.com/

MATHPIX_APP_ID=your-mathpix-app-id

MATHPIX_APP_KEY=your-mathpix-app-key

```

Verify Installation

python extractor.py --help

Get API Keys

• Anthropic Claude: https://console.anthropic.com/
• OpenAI: https://platform.openai.com/api-keys
• Mathpix OCR: https://api.mathpix.com/

How to Use

When users request data extraction:

1. Understand requirements: Ask what type of data to extract
2. Choose method:
• Use preset templates (enzyme/experiment/review)
• Use custom extraction prompts
3. Execute extraction:

```bash

python extractor.py input.pdf --template enzyme -o output.md

```

1. Verify results: Display extracted data and ask if adjustments needed

Preset Templates

Enzyme Kinetics Data (enzyme)

Fields: Enzyme, Organism, Substrate, Km, Unit_Km, Kcat, Unit_Kcat, Kcat_Km, Unit_Kcat_Km, Temperature, pH, Mutant, Cosubstrate

Experimental Results Data (experiment)

Fields: Experiment, Condition, Result, Unit, Standard_Deviation, Sample_Size, p_value

Literature Review Data (review)

Fields: Author, Year, Journal, Title, DOI, Key_Findings, Methodology

Configuration Requirements

Users should set environment variables (optional, can also be in .env file):

• EXTRACTOR_API_KEY: LLM API key
• EXTRACTOR_BASE_URL: API endpoint
• EXTRACTOR_MODEL: Model name (default: claude-sonnet-4-5-20250929)
• EXTRACTOR_TEMPERATURE: Temperature parameter (default: 0.1)
• EXTRACTOR_MAX_TOKENS: Maximum output tokens (default: 16384)
• MATHPIX_APP_ID: Mathpix OCR App ID (optional)
• MATHPIX_APP_KEY: Mathpix OCR Key (optional)

Best Practices

1. Verify API key configuration before extraction
2. Recommend users validate extracted data for accuracy
3. Long documents may require segmented processing
4. Remind users to cite original literature

Usage Examples

Example command for enzyme kinetics extraction:

python extractor.py paper.pdf --template enzyme -o results.md

Example for custom extraction:

python extractor.py paper.pdf -p "Extract all protein structures with PDB IDs" -o custom.md

Example for CSV output:

python extractor.py paper.pdf --template enzyme -o results.csv --format csv

Notes

• This tool is for academic research use only
• Always validate AI-extracted results
• Respect copyright when using extracted data
• Cite original sources appropriately