VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation

We study reward models for long-horizon manipulation by learning from action-free videos and language instructions, which we term the visual-instruction correlation (VIC) problem. Existing VIC methods face challenges in learning rewards for longhorizon tasks due to their lack of sub-stage awareness, difficulty in modeling task complexities, and inadequate object state estimation. To address these challenges, we introduce VICtoR, a novel hierarchical VIC reward model capable of providing effective reward signals for long-horizon manipulation tasks. Trained solely on primitive motion demonstrations, VICtoR effectively provides precise reward signals for long-horizon tasks by assessing task progress at various stages using a novel stage detector and motion progress evaluator. We conducted extensive experiments in both simulated and real-world datasets. The results suggest that VICtoR outperformed the best existing methods, achieving a 43% improvement in success rates for long-horizon tasks.

VICtoR employs a hierarchical approach to assess task progress at various levels, including stage, motion, and motion progress. It consists of three main components: (1) a Task Knowledge Generator that decomposes the task into stages and identifies the necessary object states and motions for each stage; (2) a Stage Detector that detects object states to determine the current stage based on the generated knowledge; (3) a Motion Progress Evaluator that assesses motion completion within stages. With this information, VICtoR then transforms it into rewards. Both the Stage Detector and Motion Progress Evaluator are trained on motion-level videos labeled with object states, which are autonomously annotated during video collection. This setup enables VICtoR to deliver precise reward signals for complex, unseen long-horizon tasks composed of these motions in any sequence.

Training Objectives: In VICtoR, two components require training: the Object Status Classifier (in the Stage Detector) and the Motion Progress Evaluator. The former is trained using cross-entropy loss, while the latter is trained with three variants of InfoNCE loss. Below are their details and schematic diagrams:

• Time Contrastive Loss $L_{tcn}$ : It encourages images that are temporally closer to have more similar representations (embeddings) than those that are temporally distant or from different videos.
• Motion Contrastive Loss $L_{mcn}$ : The objective aligns each motion's embedding with its relevant language embedding and separates it from unrelated language embeddings.
• Language-Frame Contrastive Loss $L_{lfcn}$ : It brings the progress embeddings of nearly completed steps closer to the instruction embedding of the motion while distancing the progress embeddings of frames from earlier steps.

Tasks and Baselines: To assess VICtoR's effectiveness, we train the same RL method with each reward model for every task. We construct nine simulated long-horizon manipulation tasks in CoppeliaSim and additionally evaluate all reward models on the real-world benchmark XSkill. For baselines, we compare VICtoR with the following reward models:

• Sparse Reward: A binary reward function assigns a reward only when the task succeeds.
• Stage Reward: A reward function that assigns a reward equal to the stage number when the agent reaches a new stage.
• LOReL (CoRL'21): A language-conditioned reward model that learns a classifier to evaluate whether the progression between frames at time 0 and time t aligns with the task instruction.
• LIV (ICML'23): A vision-language representation for robotics that can be utilized as a reward model by finetuning on target-domain data.

In our main experiment, a PPO policy is trained using the rewards generated by VICtoR and four other reward models for each task. The "S/M" below the task IDs represents the total number of stages and motions required to complete the task. The results clearly show that the RL method trained with VICtoR's rewards can learn to perform more complex and long-horizon tasks compared to those trained with other reward models. Notably, for long-horizon tasks, VICtoR achieves an average performance improvement of 43%.

To verify whether VICtoR provides informative rewards, we visualized the potential (reward) curves for two cases: one with videos that match the corresponding instructions and another with incorrect videos using the same instructions. For the correct action case (left), the potential curves show that VICtoR effectively identifies task progress by increasing the potential as the agent completes the task—a capability not matched by previous VIC reward models. For the incorrect action case (right), as the agent moves from the right to the left side to close the drawer, VICtoR's rewards initially show an increase in potential. This increase is logical, as these movements align with the first stage of the instruction, open the light. However, as the agent continues toward the drawer, VICtoR recognizes the incorrect task and begins to decrease the reward, effectively discouraging the agent’s movement. These two cases highlight VICtoR's ability to analyze agent movement and task progress accurately.