Video BGM Skill

Analyze a video's content and mood, generate matching BGM via AI, and mix it in.

Pipeline

Input Video → Gemini Analysis → Lyria2 BGM → FFmpeg Mix → Output
                (mood/style)     (generate)    (speed + volume + fade)

Dependencies

• Python: python3 (or activate your project venv if available)
• Gemini API: GOOGLE_GENAI_API_KEY (video understanding)
• fal.ai: FAL_API_KEY (Lyria2 music generation)
• FFmpeg: system install

Setup

# Install required packages
pip install google-generativeai httpx

# Set API keys via environment variables
export GOOGLE_GENAI_API_KEY="your_google_api_key"
export FAL_API_KEY="your_fal_api_key"

Usage

/video-bgm <path-to-video>
/video-bgm <path-to-video> --speed 1.1
/video-bgm <path-to-video> --speed 1.1 --volume 5
/video-bgm <path-to-video> --style "lo-fi chill"

Arguments

Step-by-Step Process

Step 1: Analyze Video with Gemini

Upload video to Gemini 2.0 Flash and get deep mood analysis.

import google.generativeai as genai
import os

GOOGLE_API_KEY = os.environ.get("GOOGLE_GENAI_API_KEY")
genai.configure(api_key=GOOGLE_API_KEY)

Use this analysis prompt (acts as music supervisor, not generic):

Watch this video very carefully. You are a music supervisor for commercials.

Tell me:
1. What BRAND POSITIONING does this video convey? (luxury? affordable? aspirational?)
2. What is the EMOTIONAL JOURNEY of the viewer? Be specific at each moment.
3. What REAL commercial music references would fit? Name specific ad styles
   (Four Seasons resort? Apple reveal? Volvo? Pottery Barn? Nike?)
4. What is the ENERGY LEVEL? Contemplative/still or forward momentum?
5. What tempo, instruments, and production style would ACTUALLY work?
   Be honest - classical piano is often too stuffy. Consider modern alternatives.

Then provide a SINGLE music generation prompt (2-3 sentences) that captures
the ideal BGM. Focus on: instruments, tempo BPM, mood adjectives, production style.
Format: MUSIC_PROMPT: <your prompt here>

Step 2: Extract the MUSIC_PROMPT

Parse the Gemini response to find the MUSIC_PROMPT: line. This becomes the Lyria2 prompt.

Always APPEND these constraints to any Lyria2 prompt:

No vocals, no drums, no percussion hits, no sound effects.

Step 3: Generate BGM via fal.ai Lyria2

import httpx
import os

FAL_API_KEY = os.environ.get("FAL_API_KEY")

resp = httpx.post(
    "https://fal.run/fal-ai/lyria2",
    headers={
        "Authorization": f"Key {FAL_API_KEY}",
        "Content-Type": "application/json",
    },
    json={"prompt": music_prompt},
    timeout=120.0,
)
audio_url = resp.json()["audio"]["url"]

Lyria2 generates ~32s of audio. Output is WAV, 48kHz stereo.

Step 4: FFmpeg — Speed + Strip Audio + Mix BGM

# Step 4a: Speed up video (if requested) and strip any existing audio
ffmpeg -y -i INPUT.mp4 \
  -filter:v "setpts=PTS/{speed}" \
  -an \
  -c:v libx264 -preset medium -crf 18 \
  OUTPUT_speedup.mp4

# Step 4b: Get sped-up duration
DURATION=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 OUTPUT_speedup.mp4)

# Step 4c: Mix BGM with volume boost, fade in/out, trim to video length
ffmpeg -y \
  -i OUTPUT_speedup.mp4 \
  -i bgm.wav \
  -filter_complex "[1:a]volume={volume},atrim=0:{duration},afade=t=in:st=0:d={fade_in},afade=t=out:st={duration-fade_out}:d={fade_out}[a]" \
  -map 0:v -map "[a]" \
  -c:v copy -c:a aac -b:a 192k \
  -shortest \
  OUTPUT_final.mp4

Step 5: Open Result

Open the final video for user review. Also open the BGM separately so they can evaluate the music alone.

Key Learnings

• Lyria2 output is VERY QUIET — always use volume multiplier (default 5.0)
• Don't over-specify the Lyria2 prompt — "three acts" style prompts produce chaotic results. Keep it to instruments + tempo + mood + style reference.
• Let Gemini act as music supervisor — it gives much better style recommendations than generic "relaxing piano" defaults
• Always strip existing audio first (-an) — some videos have unwanted audio tracks
• Classical piano is usually wrong — for luxury/lifestyle, modern minimalist (Rhodes, guitar, cello) works better

Output Files

All files are saved next to the input video:

input_video.mp4          → original
input_video_speedup.mp4  → sped up, no audio
input_video_bgm.wav      → generated BGM
input_video_final.mp4    → final output with BGM

Examples

# Basic: analyze and add BGM
/video-bgm ~/Desktop/product_video.mp4

# Speed up 10% and add BGM
/video-bgm ~/Desktop/product_video.mp4 --speed 1.1

# Override style (skip Gemini analysis)
/video-bgm ~/Desktop/ad.mp4 --style "upbeat modern pop, synth pads, 100 BPM"

# Adjust volume
/video-bgm ~/Desktop/quiet_video.mp4 --volume 8

Author

Canlah AI — Run performance marketing without breaking your brand.

• GitHub: github.com/PHY041
• All Skills: clawhub.ai/PHY041