A misaligned grasp is not an instantaneous event: it begins with a subtle offset during approach, escalates when the gripper closes on the wrong surface, and ends with the object slipping from the hand. Capturing that span — and explaining it — is what a recovery policy or a human operator actually needs. Prior work gives only an anomaly score, a post-hoc episode summary, or a frame-level label; we answer when a failure occurs and why, in a single streaming pass.

Comparison with related work

TG = temporal grounding. ▲ = partial (e.g. only a few sampled frames rather than full video).

A general video-language model knows little about manipulation, and no public dataset pairs time intervals with robot-action descriptions. So we generate that supervision: a multi-stage VLM pipeline turns unlabeled DROID episodes into temporally grounded captions for pre-training.

Three sequential stages turn an unlabeled DROID episode into grounded robot-action descriptions.

• Stage 1 — Summary. Enumerate the tasks in the episode as an ordered list, anchored by task recognition, an active-object trace, and a gripper state chain that together suppress hallucinated objects and actions.
• Stage 2 — Task segmentation. Localize each task as a time interval and describe it.
• Stage 3 — Phase segmentation. Split each task clip into manipulation phases from a fixed taxonomy — exploring, approaching, aligning, contacting, releasing, resetting — yielding tuples $(s, e, l, p)$.

Every stage is verified by a second VLM running on the opposite backbone from the generator (Qwen3.5-VL-122B and Cosmos-Reason2-8B), so the two models cross-check each other. Over the full DROID dataset this processes 95K videos into roughly 950K phase tuples — 1.8 tasks per video, 5.61 phases per task.

Video arrives in chunks and is encoded by the VL3 vision encoder into per-layer FIFO memory at fixed cost. A future-prediction branch raises an early alarm and the start s; the proposal heads commit the end e; the language head then decodes the description l from the window features over [s, e].

Given a stream, the model may condition only on frames up to the current step, and must output a tuple $(s, e, l)$ per failure. We build on OnVTG's streaming proposal framework but replace its visual backbone with that of Video-LLaMA3 (VL3) and reuse the paired language model, so the architecture inherits VL3's video understanding and language generation while keeping online grounding. Grounding runs on every incoming chunk for low latency; the language head runs only once an interval is committed, decoding from the localized visual features rather than from boundary tokens.

No existing resource pairs failure boundaries with causal descriptions on real robots, so we annotate one by hand on top of DROID, which holds ~16,000 unsuccessful trajectories. For each event, the start is the earliest moment the cause becomes observable (the gripper making unstable contact with a spoon) and the end is the moment the outcome is apparent (the spoon hitting the floor); each description likewise separates outcome from cause, so the annotations support causal analysis and not just detection.

The held-out split used throughout the results carries 1,212 episodes with 1,466 failure events, plus 180 verified failure-free episodes on which any predicted interval counts as a false positive, and is disjoint by trajectory ID from all training data.

Localization is scored by mean temporal IoU and F1 at IoU thresholds. Description quality is measured only on events localized correctly at IoU 0.7, so a fluent description of a mistimed detection cannot inflate the score.

Architecture comparison (online, 5 seeds)

Both fine-tuned on identical data, so architecture is the only difference. Swapping in a video-language backbone and adding a language head gains the description OnVTG cannot produce, at no cost to localization: mIoU is higher, the F1@0.7 gap of 0.003 sits inside either model's seed spread, and on the failure-free episodes STREAM-FAIL fires slightly less often (1.44 vs 1.60 spurious detections per episode).

Effect of pre-training on the generated corpus (offline, before any fine-tuning)

Pre-trained on the generated corpus alone, which contains no failure annotations. Without it, a general video-language model produces no usable descriptions (CIDEr ≤ 0.004). The gain is largest on VL3-7B, the backbone we deploy: mIoU improves by 73% and description quality rises from zero.

The corpus also helps once human labels are available: mixing it into the fine-tuning recipe improves every metric on the benchmark (mIoU 0.2365 → 0.2471, CIDEr@0.7 1.006 → 1.182, offline). Human annotation supplies precise boundaries and reference wording; the generated corpus supplies the same kind of supervision at a scale manual effort cannot reach.

Absolute reliability is still modest — mIoU 0.228, 1.44 spurious detections per failure-free episode, a mean judge score of 2.56 out of 5 — so STREAM-FAIL should be treated as an aid that flags and explains candidate failures, not as a safety guarantee. The evaluation also covers only DROID manipulation trajectories, and the benchmark relies on costly human annotation. Closing the loop with a recovery or vision-language-action policy, and a larger real-robot evaluation, are the natural next steps.