Abstract—Field robotics applications have some unique and unusual data requirements — the curating, organisation and management of which are often overlooked. An emerging theme is the use of large corpora of spatiotemporally indexed sensor data which must be searched and leveraged both offline and online. Increasingly we build systems that must never stop learning. Every sortie requires swift, intelligent read-access to gigabytes of memories and the ability to augment the totality of stored experiences by writing new memories. This however leads to vast quantities of data which quickly become unmanageable, especially when we want to find what is relevant to our needs. The current paradigm of collecting data for specific purposes and storing them in ad-hoc ways will not scale to meet this challenge. In this paper we present the design and implementation of a data management framework that is capable of dealing with large datasets and provides functionality required by many offline and online robotics applications. We systematically identify the data requirements of these applications and design a relational database that is capable of meeting their demands. We describe and demonstrate how we use the system to manage over 50TB of data collected over a period of 4 years.

Overview of the concrete data model structure. At the lowest level, raw data are stored in sequential records in flat `monolithic’ files. An index table in the DBMS holds offsets that point to these records (represented by black arrows). Annotations exist in separate tables with each individual annotation pointing to some record in the index (red arrow) and optionally to any other data — in this example, to locations in OpenStreetMap (blue arrows).


  • [PDF] P. Nelson, C. Linegar, and P. Newman, “Building, Curating, and Querying Large-scale Data Repositories for Field Robotics Applications,” in International Conference on Field and Service Robotics (FSR), Toronto, ON, Canada, 2015.
    author = {Nelson, Peter and Linegar, Chris and Newman, Paul},
    title = {{B}uilding, {C}urating, and {Q}uerying {L}arge-scale {D}ata {R}epositories for {F}ield {R}obotics {A}pplications},
    booktitle = {{I}nternational {C}onference on {F}ield and {S}ervice {R}obotics ({FSR})},
    year = {2015},
    address = {Toronto, ON, Canada},
    month = {June},
    pdf = {http://www.robots.ox.ac.uk/~mobile/Papers/2015FSR_Nelson.pdf},