Item

Abstract

Motor primitives based on dynamical systems [Ijspeert et al., 2002a] have enabled robots to learn complex tasks ranging from tennis-swings to legged locomotion. However, most interesting motor learning problems are high-dimensional reinforcement learning problems
often beyond the reach of current methods. We extend previous work [Peters and Schaal, 2006b] on policy learning from the immediate reward case to episodic reinforcement learning. We present a novel algorithm for policy learning by assuming a form of exploration that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable in complex motor learning tasks. We compare this
algorithm to several well-known parametrized policy search methods and show that it
outperforms them. We apply it in the context of motor learning and show that it can learn
a complex Ball-in-a-Cup task using a real Barrett WAM TM robot arm.
The learned open loop policy trajectory can be very sensitive to perturbations of the
initial conditions or the trajectory. Perceptual coupling is a natural choice to cancel these perturbations. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus [Pongas et al., 2005], and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. We propose an augmented version of the motor primitives based on dynamical systems which incorporates perceptual coupling to an external variable.
The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such as Ball-in-a-Cup even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using our novel reinforcement learning method.