Dexterous manipulation of objects is a core task in robotics. Because of the design complexity needed for robot controllers even for simple manipulation tasks for humans, e.g., pouring a cup of tea, robots currently in use are mostly limited to specific tasks within a known environment. While humans learn their dexterity over time and manipulate objects through dynamic hand-eye coordination using visuo-tactile feedback [Johansson&Randall Flanagan2009], most recent research work on robotic manipulation is data-driven, primarily based on visual perception, to learn a one shot manipulation model which cannot generalize when objects or environments come to change [Bohg et al.2014, Mousavian et al.2019].

In this PhD thesis, we aim to learn a general purpose robot multi-task controller to solve instruction-conditioned manipulation tasks from vision, e.g., Pick Object, Pass Object.