LLM Inference Performance Estimator

Estimate TTFT (Time To First Token), decode speed (tokens/s), and VRAM usage for a given LLM on a specific GPU.

How to Use

The user may invoke this skill in several ways:

1. Named model: /llm-perf-estimator Qwen2.5-7B RTX4090 2048 512 fp16
2. With config file: /llm-perf-estimator config.json RTX4090 2048 512 int4
3. Interactive: /llm-perf-estimator — ask the user step by step

Arguments (all optional, prompt for missing ones):

• model — model name from preset list, or path to a HuggingFace config.json
• gpu — GPU name from preset list, or custom specs
• input_tokens — prefill sequence length (default: 1024)
• output_tokens — number of tokens to generate (default: 256)
• quant — quantization format: fp16, bf16, fp8, int8, int4 (default: fp16)

Step 1 — Resolve Model Architecture

Preset Models

If the user provides a known model name, use the following presets:

| Model | Type | Total Params | Activated Params | Layers | Hidden | Heads (Q) | Heads (KV) | FFN Type | Intermediate | Vocab |

|---|---|---|---|---|---|---|---|---|---|---|

| Qwen3.5-4B | Hybrid Dense | 4B | 4B | 32 (8 full+24 linear) | 2560 | 16 (full) / 16 (linear) | 4 (full) | SwiGLU | 9216 | 248320 |

| Qwen3.5-35B-A3B | Hybrid MoE | 35B | 3B | 40 (10 full+30 linear) | 2048 | 16 (full) / 16 (linear) | 2 (full) | SwiGLU+MoE | 8×512 per tok | 248320 |

If the model is not in the preset list and no config file is provided, ask the user to provide a config.json. They can get it without downloading the full model:

# ModelScope (browser)
https://modelscope.cn/models/{org}/{model}/file/view/master/config.json

# HuggingFace (browser)
https://huggingface.co/{org}/{model}/blob/main/config.json

Open the URL, copy the content, and paste it directly into the conversation. Alternatively, provide the local file path if the model is already downloaded.

If the user cannot provide a config, ask them to manually input:

• num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads
• intermediate_size, vocab_size
• For MoE: num_experts, num_experts_per_tok, moe_intermediate_size

Parsing config.json

If the user provides a config.json path, read the file and extract:

num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads,
intermediate_size, vocab_size, model_type,
# MoE fields (if present):
num_experts / num_local_experts, num_experts_per_tok, moe_intermediate_size
# Hybrid attention (if present):
layer_types  ← list of strings, e.g. ["linear_attention", ..., "full_attention", ...]
head_dim     ← if explicitly provided, use it; otherwise head_dim = hidden_size / num_attention_heads

Determine num_full_attn_layers:

• If layer_types exists: num_full_attn_layers = count of "full_attention" in layer_types
• If layer_types is absent (standard transformer): num_full_attn_layers = num_hidden_layers

Note on nested configs (e.g. Qwen3.5-35B-A3B has a text_config wrapper):

• If the top-level JSON has a text_config key, read all text model fields from inside it.
• head_dim may be explicitly set (e.g. 256); prefer that over computing from hidden_size / num_attention_heads.

Note on tie_word_embeddings: if true, the embedding table and lm_head share the same weights. Do not count them twice in VRAM — the embedding contributes vocab_size × hidden_size × bytes_per_param only once.

Note on attn_output_gate: recognized but ignored in calculations — its contribution to FLOPs and VRAM is <1% and within the MFU uncertainty margin.

Step 2 — Resolve GPU Specs

Preset GPUs

| GPU | VRAM (GB) | BF16 TFLOPS | FP8 TFLOPS | INT8 TOPS | HBM BW (GB/s) |

|---|---|---|---|---|---|

| RTX 4060 | 8 | 15.1 | — | 30.2 | 272 |

| RTX 4060 Ti | 16 | 22.1 | — | 44.2 | 288 |

| RTX 4070 | 12 | 29.1 | — | 58.2 | 504 |

| RTX 4070 Ti | 12 | 40.1 | — | 80.2 | 504 |

| RTX 4070 Ti Super | 16 | 40.1 | — | 80.2 | 672 |

| RTX 4080 | 16 | 48.7 | — | 97.4 | 717 |

| RTX 4080 Super | 16 | 52.2 | — | 104.4 | 736 |

| RTX 4090 | 24 | 82.6 | — | 165.2 | 1008 |

| RTX 5070 Ti | 16 | 176.0 | 352.0 | 352.0 | 896 |

| RTX 5080 | 16 | 225.0 | 450.0 | 450.0 | 960 |

| RTX 5090 | 32 | 419.0 | 838.0 | 838.0 | 1792 |

| A10G | 24 | 31.2 | — | 62.5 | 600 |

| A100-40G | 40 | 77.97 | — | 311.9 | 1555 |

| A100-80G | 80 | 77.97 | — | 311.9 | 2000 |

| H100-SXM | 80 | 989.4 | 1978.9 | 3958.0 | 3350 |

| H100-PCIe | 80 | 756.0 | 1513.0 | 3026.0 | 2000 |

| H200-SXM | 141 | 989.4 | 1978.9 | 3958.0 | 4800 |

| L4 | 24 | 30.3 | 60.6 | 121.2 | 300 |

| L40S | 48 | 91.6 | 183.2 | 366.4 | 864 |

| MI300X | 192 | 1307.4 | 2614.9 | 5229.8 | 5300 |

| Apple M4 (16GB) | 16 | 4.6 | — | — | 120 |

| Apple M4 Pro (48GB) | 48 | 9.2 | — | — | 273 |

| Apple M4 Max (128GB) | 128 | 18.4 | — | — | 546 |

If the GPU is not listed, ask the user to provide:

• VRAM (GB)
• BF16/FP16 TFLOPS
• HBM bandwidth (GB/s)

Step 3 — Quantization Bytes Per Parameter

| Format | Bytes/param | Compute dtype | Notes |

|---|---|---|---|

| fp32 | 4.0 | fp32 | Rarely used for inference |

| bf16 / fp16 | 2.0 | bf16/fp16 | Baseline |

| fp8 | 1.0 | fp8 | Requires H100/H200/RTX50xx |

| int8 | 1.0 | int8 | W8A8 or W8A16 |

| int4 | 0.5 | int4/fp16 | GPTQ/AWQ/bitsandbytes |

Select the GPU TFLOPS column matching the compute dtype:

• fp16/bf16 → BF16 TFLOPS
• fp8 → FP8 TFLOPS (fall back to BF16 if not supported, with a warning)
• int8 → INT8 TOPS
• int4 → BF16 TFLOPS (dequant to fp16 for matmul in most frameworks)

Step 4 — Compute VRAM Requirements

4.1 Weight Memory

weight_bytes = total_params × bytes_per_param
weight_GB = weight_bytes / 1e9

For MoE models, total_params includes all expert weights (not just activated).

4.2 KV Cache Memory

Only full attention layers maintain a KV cache. Linear attention layers use a fixed-size recurrent state (negligible, ~tens of MB) that does not grow with sequence length.

kv_heads = num_key_value_heads          # from full attention config
kv_bytes_per_token = 2 × num_full_attn_layers × kv_heads × head_dim × bytes_per_param
kv_cache_GB = kv_bytes_per_token × (input_tokens + output_tokens) / 1e9

If num_full_attn_layers = num_hidden_layers (standard transformer), this reduces to the standard formula.

4.3 Activation Memory (prefill peak)

activation_GB ≈ num_layers × hidden_size × input_tokens × bytes_per_param × 2 / 1e9

This is an approximation; actual peak depends on framework and attention implementation.

4.4 Total VRAM

total_VRAM_GB = weight_GB + kv_cache_GB + activation_GB

Add a 15% overhead for framework buffers, CUDA context, etc.:

total_VRAM_GB_with_overhead = total_VRAM_GB × 1.15

Step 5 — Estimate TTFT (Prefill Latency)

Prefill is compute-bound for long sequences.

5.1 Attention FLOPs (prefill)

Only full attention layers have O(n²) attention compute. Linear attention layers are O(n) and their attention FLOPs are already captured in the projection FLOPs (Step 5.3).

attn_flops = 4 × num_full_attn_layers × input_tokens² × hidden_size

(factor of 4 = QK matmul + softmax + AV matmul, forward pass)

If num_full_attn_layers = num_hidden_layers, this is the standard transformer formula.

5.2 FFN FLOPs (prefill)

For SwiGLU/GeGLU (3 projections: gate, up, down):

ffn_flops = 3 × 2 × num_layers × input_tokens × hidden_size × intermediate_size

For MoE, replace intermediate_size with num_experts_per_tok × moe_intermediate_size.

5.3 QKV + Output Projection FLOPs

For full attention layers (standard QKV projections):

full_proj_flops = 2 × num_full_attn_layers × input_tokens × hidden_size
                  × (num_attention_heads × head_dim + 2 × kv_heads × head_dim + hidden_size)

For linear attention layers (also have Q/K/V-equivalent projections, but different dims):

linear_proj_flops = 2 × num_linear_attn_layers × input_tokens × hidden_size
                    × (linear_num_key_heads × linear_key_head_dim
                       + linear_num_key_heads × linear_key_head_dim
                       + linear_num_value_heads × linear_value_head_dim
                       + hidden_size)

If layer_types is absent (standard transformer), only full_proj_flops applies and num_linear_attn_layers = 0.

5.4 Total Prefill FLOPs

total_prefill_flops = attn_flops + ffn_flops + full_proj_flops + linear_proj_flops

5.5 TTFT

Apply MFU (Model FLOP Utilization) efficiency factor:

| Scenario | MFU |

|---|---|

| Long prompt (>512 tokens), data center GPU | 0.45 |

| Long prompt, consumer GPU | 0.35 |

| Short prompt (<128 tokens) | 0.25 |

effective_tflops = gpu_tflops × MFU
TTFT_seconds = total_prefill_flops / (effective_tflops × 1e12)

Step 6 — Estimate Decode Speed

Decode is memory-bandwidth-bound at batch=1.

6.1 Bytes Read Per Decode Step

Each decode step reads:

• All activated model weights once
• KV cache for all previous tokens (full attention layers only; linear attention state is fixed-size and already loaded with weights)

activated_weight_bytes = activated_params × bytes_per_param
kv_cache_bytes_at_step = kv_bytes_per_token × (input_tokens + current_output_tokens)
bytes_per_step = activated_weight_bytes + kv_cache_bytes_at_step

For the average decode step, use current_output_tokens ≈ output_tokens / 2.

6.2 Decode Speed

Apply bandwidth utilization efficiency factor:

| Scenario | BW Utilization |

|---|---|

| Data center GPU (HBM2e/HBM3) | 0.85 |

| Consumer GPU (GDDR6X) | 0.75 |

| Apple Silicon (unified memory) | 0.80 |

effective_bandwidth = gpu_bandwidth_GBs × bw_utilization
decode_speed_tps = effective_bandwidth × 1e9 / bytes_per_step

Step 7 — Output Report

Present results as a Markdown report with the following sections:

Section 1: Configuration Summary

| Parameter | Value |

|---|---|

| Model | {model_name} |

| Type | Dense / MoE / Hybrid MoE |

| Total Params | {X}B |

| Activated Params | {X}B |

| Total Layers | {N} |

| Full Attention Layers | {N} ({N} linear attention) |

| GPU | {gpu_name} |

| VRAM Available | {X} GB |

| Quantization | {quant} |

| Input Tokens | {N} |

| Output Tokens | {N} |

Section 2: VRAM Breakdown

| Component | Size (GB) |

|---|---|

| Model Weights | {X} |

| KV Cache | {X} |

| Activations (peak) | {X} |

| Framework Overhead (15%) | {X} |

| Total Required | {X} |

| GPU Available | {X} |

| Fits in VRAM? | ✅ Yes / ❌ No |

If it doesn't fit, suggest:

• A lower quantization format
• Offloading options (CPU offload, disk offload)

Section 3: Performance Estimates

| Metric | Estimate |

|---|---|

| TTFT (Time to First Token) | {X} ms |

| Decode Speed | {X} tokens/s |

| Time to Generate {N} tokens | {X} s |

| Total End-to-End Latency | {X} s |

Section 4: Assumptions & Caveats

List the MFU and bandwidth utilization values used, and note:

• Estimates assume batch_size=1, single GPU
• Actual performance varies by framework (vLLM, llama.cpp, Ollama, etc.)
• FlashAttention / FlashAttention-2 is assumed for prefill
• KV cache quantization not considered
• Speculative decoding not considered

Notes for the Agent

• Always show intermediate calculations in a collapsible section or footnote if the user asks "how did you calculate this"
• If VRAM is insufficient, proactively suggest the minimum quantization that would fit
• If the user provides a config.json, confirm the parsed values before computing
• Round all results to 2 significant figures for readability
• For MoE models, clearly distinguish total vs activated parameters in all calculations