Data Analyzer — 数据分析工具

Load, analyze, and report on structured data from CSV, Excel, and JSON files. Compute statistics, detect anomalies, identify trends, and generate reports with visualization recommendations.

Workflow

1. Load data     → Read the file, inspect structure
2. Profile       → Column types, missing values, basic stats
3. Analyze       → Statistics, trends, anomalies, correlations
4. Report        → Summary with visual recommendations

Step 1 — Data Loading

Supported Formats

If Python is available (recommended for large datasets):

pip install pandas openpyxl  # if missing
python3 -c "
import pandas as pd
df = pd.read_csv('data.csv')
print(df.info())
print(df.describe())
print(df.head())
"

If Python is not available, parse manually:

1. Read the file line by line
2. Identify headers (first row)
3. Identify column types (numeric vs text vs date)
4. Store as an array of rows or objects

Initial Inspection

After loading, always answer these questions:

• Shape: How many rows and columns?
• Column names: What are they and what data types?
• Missing values: Which columns have gaps, and how many?
• Date/time columns: Are they parsed as datetime objects?
• Unique values: For categorical columns, how many unique categories?

Step 2 — Descriptive Statistics

Numeric Columns

Compute and report:

Formula reference (manual calculation):

Mean       = sum(x) / n
Median     = middle value when sorted
Std Dev    = sqrt(sum((x - mean)^2) / (n-1))
Percentile = sort values, take value at position (p/100 * n)

Categorical Columns

Step 3 — Analysis

3a. Anomaly Detection

Method: IQR (Interquartile Range)

Q1 = 25th percentile
Q3 = 75th percentile
IQR = Q3 - Q1
Lower fence = Q1 - 1.5 * IQR
Upper fence = Q3 + 1.5 * IQR
Anomaly = any value outside [Lower fence, Upper fence]

Method: Z-Score (for approximately normal distributions)

z = (x - mean) / std_dev
Anomaly = |z| > 3 (values more than 3 std devs from mean)

Output anomalies: For each detected anomaly, report:

• Row index
• Column name
• Anomalous value
• Distance from expected (how many IQRs or std devs)

3b. Trend Analysis

For time-series data (data with a date/time column):

1. Identify the time column — Sort by date
2. Aggregate by period — Group by day/week/month/quarter/year
3. Direction — Is the metric increasing, decreasing, or flat?
4. Rate of change — Period-over-period percentage change
5. Seasonality — Recurring patterns (monthly, quarterly, yearly)
6. Breakout — Sudden jumps or drops (potential regime changes)

Output format:

📈 Trend: [Metric Name]
Period: [Date Range]
Direction: [Up/Down/Flat] (slope: ±X%)
Key Points:
- [Date]: Value = X (↗/↘/→)
- Highest point: [Date] = X
- Lowest point: [Date] = X

For non-time-series data, analyze rank order and distribution shape:

Top 5 by [metric]:
1. [Category] = X
2. [Category] = Y
...
Bottom 5 by [metric]:

3c. Correlation Analysis

Pearson correlation coefficient (for linear relationships between two numeric variables):

r = sum((x - mean_x) * (y - mean_y)) / (n * std_x * std_y)

Interpretation:

Caveats:

• Correlation ≠ causation. Always note this.
• Pearson only captures linear relationships.
• Outliers can distort correlation heavily — check after removing anomalies.

Step 4 — Report Generation

Visualization Recommendations

For each finding, recommend the best chart type:

Full Report Template

# Data Analysis Report: [Dataset Name]
Date: [YYYY-MM-DD]

## 1. Overview
- Rows: X | Columns: Y
- Missing data: X cells (X%)
- Key columns: [list with types]

## 2. Descriptive Statistics
### Numeric Columns
[Table: col_name, count, mean, median, std, min, 25%, 75%, max]

### Categorical Columns
[Table: col_name, unique_count, top_value, frequency%]

## 3. Key Findings

### Finding 1: [Title]
[Description of finding]
📊 Recommended chart: [Chart type]
Supporting data: [stats/view]

### Finding 2: [Title]
...

## 4. Anomalies Detected
[Table: row, column, value, severity]

## 5. Correlations
[Notable correlations >|0.3| or < -|0.3|]

## 6. Recommendations
[Data-driven suggestions based on analysis]

One-Page Summary (Quick)

For quick results, use this compact format:

📊 [Dataset]: [N] rows × [M] cols

📈 Key metrics:
- [metric1]: mean=X, median=Y, range=[min, max]
- [metric2]: ...

🔍 Top findings:
1. [Finding] — [chart recommendation]
2. [Finding] — [chart recommendation]

⚠️ Anomalies: X detected

Python Script (Optional)

For complex analysis, create and run a Python script:

import csv, json, statistics
from collections import Counter

# Load data
with open('data.csv') as f:
    reader = csv.DictReader(f)
    rows = list(reader)

# Get numeric columns
# (column name → list of float values, filtering out blanks)
# Compute mean, median, stdev, percentiles
# Detect outliers via IQR
# Compute correlations between pairs
# Print formatted results

Run with:

python3 analysis.py