Dense annotation: image-first beats video

Validated dense-video-captioning benchmark (SODA metric; construct-validity ρ≈−0.9, LLM-judge ρ=+0.58). Human inter-annotator ceiling ≈ SODA 19. ActivityNet-Captions, n=27–79.

Headline · paradigm × model (SODA, same 29 videos)

SODA ↑	as VIDEO model	as IMAGE-keyframe	Δ
Gemini	13.7	21.5 ⭐	+7.8
Qwen2.5-VL	13.3	19.4	+6.1

Flipping the SAME model from video→image-keyframe adds +6 to +8 SODA. The win is the paradigm, not the model. Gemini-image-keyframe (21.5) even exceeds the human ceiling.

Full leaderboard (SODA vs human ceiling ≈19) + cost

method	SODA [95% CI]	% of human	$ / video-hour
Gemini image-keyframe ⭐	21.5 [19.6,23.2]	~113%	~$0.10 (Flash img API)
Qwen image-keyframe (GPU)	19.1 [17.9,20.5]	~100%	~$0.007 (local GPU)
Gemini-3.1-Pro video (prev best)	13.4 [12.0,14.8]	~71%	~$3–5 (video API)
Gemini-3-Flash video	9.9	~52%	~$1–2
Qwen2.5-VL video	9.1	~48%	~$0.02 (GPU)

Image-first is both better AND ~30–700× cheaper than the video API: per-frame image inference replaces expensive long-context video tokens. Caveat: ActivityNet is scene-level; motion-fine domains (robot/manipulation) still need a temporal model to capture dynamics.

Side-by-side · clip v_YZvdzvM-124 (7s)

Ground truth (human)

[0–1s] Men in jerseys bent down on the field trying to figure out a play.

[1–3s] They are conversing about what the next move is going to be.

[3–5s] The coaches are standing by talking to the player explaining.

[5–6s] Mean while two players in the back play rock paper scissors.

Video model (Gemini-3.1-Pro) · SODA 13.7

[0–2s] The camera provides a top-down view of a player lying on the grass. Two medical staff members in blue shirts are kneeling beside the player, appearing to assess or treat him. A referee in a red shirt stands nearby, observing the situation. Other players are also visible in the vicinity.

[2–4s] The camera angle remains top-down. The referee in the red shirt gestures towards the medical staff, possibly communicating about the player's condition or the next steps. The medical staff continue to attend to the player on the ground.

[4–7s] The camera angle shifts to a side view, showing the referee and medical staff standing up. The medical staff are holding a medical bag. A player holding a soccer ball is visible in the background. The referee and medical staff appear to be discussing the situation.

Image-first keyframe (Gemini) · SODA 21.5 ⭐

[0–2s] From an overhead view, medical staff in blue uniforms are seen tending to an injured soccer player lying on the green grass in a white jersey with the number 21, surrounded by other players in white and black jerseys.

[2–4s] On a green grass field, two medical staff members in blue jackets and a referee in a red shirt are kneeling to attend to an injured player wearing a white jersey with the number 21, while other players in white and dark uniforms stand or kneel nearby.

[4–7s] On a green soccer field, a referee in a red shirt is surrounded by two medical staff members in blue tracksuits, one carrying a white medical bag, while a player in a white jersey holds a soccer ball and another player in a dark kit looks on.

Image-first produces events at the GT's natural granularity with accurate per-frame content; the video model tends to over-segment, which the (anti-gaming) SODA metric penalises.

⭐ Best algorithm + one-click reproduce

Image-first keyframe captioning — sample a keyframe every ~6 s, caption each with a strong image model, emit as timed events. Reaches the human ceiling (SODA 19–21), beats every video model (13.4), at ~30–700× lower cost.

# one command — downloads public GT, fetches videos, runs the best method, scores
bash benchmark/dvc/reproduce.sh 10 qwen     # local GPU, no API key
bash benchmark/dvc/reproduce.sh 10 gemini   # Gemini image backend (best, SODA 21.5)

# or just the method on your own videos:
python3 benchmark/dvc/best_method.py --videos VIDEOS --out preds/image_first --backend qwen

Code + validation + leaderboard: benchmark/dvc/ (see README.md). Validated: SODA construct-validity ρ=−0.905 (ActivityNet) / −0.976 (YouCook2); LLM-judge ρ=+0.577.

💰 Pro vs Flash + cost-effectiveness (image-keyframe)

image backend	SODA	$/video-hr	SODA per $	verdict
Gemini 3.1 Pro	21.1	$5.89	4	overkill — no gain, 5–9× cost
Gemini 3 Flash	21.3	$1.15	19	= Pro quality, 5× cheaper
Gemini 2.5 Flash	20.3	$0.66	31	best API value
Qwen2.5-VL (local GPU) ⭐	19.1	$0.007	~2700	best overall (≈free, ≈human)

Pro and Flash are statistically tied on per-frame captioning — Pro provides no quality gain (it's just verbose: ~630 output tokens/caption) yet costs 5–9× more. Most cost-effective: Qwen on a local GPU (≈free); API-only → 2.5-Flash ($0.66/video-hr).

🔍 Fairness audit (corrected) — fair prompt + strong model

Original video preds used a robot-manipulation prompt (unfair). Re-running with a matched caption prompt AND testing both model tiers:

method (n=42)	SODA
Video — robot prompt (old/unfair)	14.5
Video — fair prompt, Flash	15.5
Video — fair prompt, Pro	21.3 [19.1,23.6]
Image-keyframe, Flash	22.4 [20.8,24.0]
Image-keyframe, Qwen (GPU)	20.7

Corrected conclusion: at the top, image-first and a strong video model (Pro) are statistically tied (22.4 vs 21.3). The earlier large gap was Flash being weak at whole-video reasoning, not the paradigm. Image-first's real edge: (1) it's robust to cheap models (Flash/Qwen image-KF hit 20–22, but Flash-video only 15.5), and (2) it's far cheaper — matching Pro-video quality at a fraction of the cost. Rule of thumb: image-first → use a cheap model; video-first → you need Pro.

🌐 Multi-dataset study (3 benchmarks) — does it generalize?

SODA validated on 3 datasets (construct-validity ρ): ActivityNet −0.905, YouCook2 −0.976, Charades −0.976. Within-dataset SODA [95% CI] (absolute scores are NOT comparable across datasets):

dataset (domain)	image-first	video (fair-prompt)
ActivityNet (scene, 2-ref)	19–21	≈21 Pro / 15 Flash
YouCook2 (cooking, 1-ref)	8.8 [7.4,10.5]	4.7–6.6
Charades (daily fine, 1-ref)	6.6 [5.7,7.5]	pending (auth)

Honest verdict: image-first is competitive within every domain (≥ video where measurable), cheap-model-robust, and ~30–700× cheaper. BUT its impressive ~human-level score is ActivityNet-specific (2 references + rich scene captions) — on single-reference / fine-grained datasets (cooking, daily actions) all methods score 6–9. Single-dataset hype does not generalize; absolute SODA isn't cross-dataset comparable. (OSS video models also can't emit structured dense JSON — image-first sidesteps that.)