THEME · Physical AI

Foundation Models

Vision-language(-action) models and LLMs that plan, act, and explain failures — and how to adapt or compress them for robots and agents.

8 projects

CLAPCoRL 2026
Direct VLM-to-VLA Adaptation via Language-Action Grounding
CLAP (Causal Language-Action Prediction) turns pretrained VLMs into VLAs by prepending a natural-language action plan to each numeric action chunk, so action prediction is causally conditioned on it with no architectural modification, action expert, or extra annotation. 2B CLAP hits 90.8% on LIBERO (+14.9 pt over VLA-0) with 4-5 hours of fine-tuning.
Online_Robot_Error_DetectionCoRL 2026
Online Robot Manipulation Error Detection and Description
We formulate robot failure detection as online video temporal grounding, predicting the temporal boundaries and a cause-and-effect description of each manipulation failure as the video streams.
dgpoACL 2026
Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities
We investigate whether compact language models (0.5–1B) can acquire sophisticated agentic retrieval-augmented generation (RAG) behavior. While existing agentic RAG systems rely on multi-billion-parameter models, compact models suffer from poor initial outputs, sparse rewards, and unstable reinforcement learning dynamics. To address these challenges, we propose Distillation-Guided Policy Optimization (DGPO), a two-phase training framework combining cold-start knowledge distillation and selective teacher-guided reinforcement learning.
ViLaIn-TAMPIROS 2026
Grounded Vision-Language Interpreter for Integrated Task and Motion Planning
While recent advances in vision-language models (VLMs) have accelerated the development of language-guided robot planners, their black-box nature often lacks safety guarantees and interpretability crucial for real-world deployment. This paper proposes ViLaIn-TAMP, a hybrid planning framework for enabling verifiable, interpretable, and autonomous robot behaviors.
compliance-enabled-failure-recoveryICRA 2026
Robust and Resilient Soft Robotic Object Insertion with Compliance-Enabled Contact Formation and Failure Recovery
By integrating VLM-guided failure recovery into compliance-enabled contact formation, we achieve robust and resilient soft robotic object insertion without force sensing or controller retraining.
visuo-tactile-recognitionIROS 2024
Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model
null
language-guided-pattern-formationIROS 2024
Language-Guided Pattern Formation for Swarm Robotics with Multi-Agent Reinforcement Learning
This paper explores how to leverage the vast knowledge encoded in large language models to tackle pattern formation challenges for swarm robotics systems.
TacZeroPreprint
Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback
A general-purpose vision-language model interprets visual and tactile observations and selects robot actions without additional tactile or manipulation training.

Direct VLM-to-VLA Adaptation via Language-Action Grounding

Online Robot Manipulation Error Detection and Description

Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities

Grounded Vision-Language Interpreter for Integrated Task and Motion Planning

Robust and Resilient Soft Robotic Object Insertion with Compliance-Enabled Contact Formation and Failure Recovery

Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model

Language-Guided Pattern Formation for Swarm Robotics with Multi-Agent Reinforcement Learning

Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback