In this PhD thesis, we aims to develop an AI-empowered general-purpose robotic system for dexterous bi-manual manipulation utilizing multimodal information from visual and tactile sensors. This will be achieved by combining novel representations with skill learning methods for learning to manipulate complex objects of different physical properties, as well as for abstracting and decomposing challenging bi-manipulation tasks into internal simpler ones. Specifically, this PhD research project will make use of dual-arm robots, e.g., TIAGo++, equipped with dexterous three-fingered hands with tactile sensing and aims to make breakthroughs in the following research topics:
1) Feature representation learning and control of compliant grippers with tactile sensing able to physically adapt to unknown object’s shape and provide feedback about object properties, e.g., grasp instability, physical properties of objects, fragility, etc.
2) Visuotactile fusion for a deep understanding of the scene, i.e., object grasp configurations prediction, in-hand pose estimation, affordances understanding, etc;
3) A hierarchical learning approach for coordinated bi-manipulation, i.e., an internal decomposition of the tasks into subtasks for two-hands-eye coordination of the dual-arm robot, e.g., for abstracting the internal phases of tasks like grasp a lamp-base and a bulb, coordinate the two pieces, put them together, and screw bulb into the base;