ILA Watch video

Interaction Latents as Actionsfor Contact-Rich Manipulation

Yuhang Zheng1,* Yujie Zang1,* Yuchen Liu2,4,* Yupeng Zheng2,3,5,+ Zhennan Jiang5 Zebin Xing3,5 Shuai Tian3,5 Songen Gu3,6 Jingxiang Guo1 Chen Gao1,+ Zhongpu Xia2,† Shuicheng Yan1,+
1 National University of Singapore 2 Anyverse Robotics 3 Open-phoenix 4 Beijing Institute of Technology 5 University of the Chinese Academy of Sciences 6 Fudan University

* Equal contribution† Project lead+ Corresponding authors

Watch video
2×
2×
2×
2×

Overview

ILA overview: visual, tactile, and motion information are encoded into a shared interaction latent space used by FM, VLA, and WAM policies and an expert-consistency reward model.

We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement.

Method

ILA method: visually conditioned joint action and tactile encoding, latent-space policy learning with a shared decoder, and offline improvement using expert-consistency rewards and compatible expert references.

  • Interaction representation

    A visually conditioned encoder maps aligned action and tactile sequences into shared interaction latents. A time-conditioned decoder reconstructs actions, tactile deformation fields, and aggregate deformation as a force proxy.

  • Latent-space policy learning

    Interaction latents serve as flow-matching targets for post-training FM, VLA, and WAM policies. The frozen shared decoder converts predicted latents into executable actions.

  • Offline policy improvement

    Dense interaction rewards guide offline policy improvement through reward-weighted learning from successful rollouts and expert-guided correction of failed rollouts.

Results

Interaction latent actions

BaselinesILA

Success rate (%) Higher is better ↑

Execution force-proxy error (Fext L1) Lower is better ↓

Compared methods. FM-tac, VLA-tac, and FastWAM-tac directly predict actions with tactile conditioning; FM-tac*, VLA-tac*, and DreamTac jointly predict actions and tactile signals. Methods within each family share expert data and evaluation conditions. Force error (Fext L1) measures the L1 difference between executed force-proxy signals and expert references.

Policy improvement

FM-ILAFM-ILA-RL

With dense interaction rewards and expert-guided correction, FM-ILA-RL improves mean success rate from 80% to 85.25% (+5.25 pp) and reduces mean execution force-proxy error from 3.39 to 1.82 (46.4% lower) compared with FM-ILA.

Demonstrations