Imagine That! Leveraging Emergent Affordances for Tool Synthesis in Reaching Tasks


In this paper we investigate an artificial agent’s ability to perform task-focused tool synthesis via imagination. Our motivation is to explore the richness of information captured by the latent space of an object-centric generative model –and how to exploit it. In particular, our approach employs activation maximisation of a task-based performance predictor to optimise the latent variable of a structured latent-space model in order to generate tool geometries appropriate for the task at hand. We evaluate our model using a novel dataset of synthetic reaching tasks inspired by the cognitive sciences and behavioural ecology. In doing so we examine the model’s ability to imagine tools for increasingly complex scenario types, beyond those seen during training. Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way: the agents often specifically modify aspects of the tools which relate to meaningful (yet implicitly learned) concepts such as a tool’s length, width and configuration. Our results therefore suggest that task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience.

This figure shows the evolution of tools during the imagination process. Each row illustrates how the imagination procedure can succeed at constructing tools that solve a task by: (A) increasing tool length, (B) decreasing tool width, and (C) changing sticks into hooks (and vice versa). Overlapping areas are marked in bright yellow. Arrows indicate optimisation steps during tool synthesis.

  • [PDF] Y. Wu, S. Kasewa, O. Groth, S. Salter, L. Sun, O. P. Jones, and I. Posner, “Imagine That! Leveraging Emergent Affordances for Tool Synthesis in Reaching Tasks,” arXiv preprint arXiv:1909.13561, 2019.
    title={Imagine That! Leveraging Emergent Affordances for Tool Synthesis in Reaching Tasks},
    author={Wu, Yizhe and Kasewa, Sudhanshu and Groth, Oliver and Salter, Sasha and Sun, Li and Jones, Oiwi Parker and Posner, Ingmar},
    journal={arXiv preprint arXiv:1909.13561},
    year = {2019},
    pdf = {}