Deep tracking in the wild: End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks

Abstract – This paper presents a novel approach for tracking static and dynamic objects for an autonomous vehicle operating in complex urban environments.
Whereas traditional approaches to tracking often feature numerous hand-engineered stages, this method is learned end-to-end and can directly predict a fully unoccluded occupancy grid from raw laser input. We employ a recurrent neural network (RNN) to capture the state and evolution of the environment, and train the model in an entirely unsupervised manner. In doing so, our use case compares to model-free, multi-object tracking although we do not explicitly perform the underlying data-association process. Further, we demonstrate that the underlying representation learned for the tracking task can be leveraged via inductive transfer to train an object detector in a data efficient manner. We motivate a number of architectural features and show the positive contribution of dilated convolutions, dynamic and static memory units to the task of tracking and classifying complex dynamic scenes through full occlusion. Our experimental results illustrate the ability of the model to track cars, buses, pedestrians, and cyclists from both moving and stationary platforms. Further, we compare and contrast the approach with a more traditional model-free multi-object tracking pipeline, demonstrating that it can more accurately predict future states of objects from current inputs.

  • [PDF] J. Dequaire, P. Ondrúška, D. Rao, D. Wang, and I. Posner, “Deep tracking in the wild: End-to-end tracking using recurrent neural networks,” The International Journal of Robotics Research, 2017.
    [Bibtex]

    @article{DequaireIJJ2017,
    author = {Dequaire, Julie and Ondr{\'u}{\v{s}}ka, Peter and Rao, Dushyant and Wang, Dominic and Posner, Ingmar},
    title = {Deep tracking in the wild: End-to-end tracking using recurrent neural networks},
    year = {2017},
    eprint = {http://journals.sagepub.com/doi/abs/10.1177/0278364917710543},
    journal = {The International Journal of Robotics Research},
    publisher={SAGE Publications Sage UK: London, England},
    Pdf = {http://www.robots.ox.ac.uk/~mobile/Papers/2017_IJRR_Dequaire.pdf}
    }

  • [PDF] P. Ondruska, J. Dequaire, D. Zeng Wang, and I. Posner, “End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks,” in Robotics: Science and Systems, Workshop on Limits and Potentials of Deep Learning in Robotics, 2016.
    Best Workshop Paper [Bibtex]

    @inproceedings{OndruskaRSS2016,
    Author = {Ondruska, Peter and Dequaire, Julie and Zeng Wang, Dominic and Posner, Ingmar},
    Title = "{End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks}",
    Booktitle = {Robotics: Science and Systems, Workshop on Limits and Potentials of Deep Learning in Robotics},
    Year = 2016,
    Month = June,
    Pdf = {http://www.robots.ox.ac.uk/~mobile/Papers/2016RSS_ondruska.pdf},
    award = "Best Workshop Paper",
    awardlink = "http://juxi.net/workshop/deep-learning-rss-2016/#papers"
    }

  • [PDF] [GITHUB] P. Ondruska and I. Posner, “Deep Tracking: Seeing Beyond Seeing Using Recurrent Neural Networks,” in The Thirtieth AAAI Conference on Artificial Intelligence (AAAI), Phoenix, Arizona USA, 2016.
    [Bibtex]

    @inproceedings{OndruskaAAAI2016,
    Address = {Phoenix, Arizona USA},
    Author = {Peter Ondruska and Ingmar Posner},
    Booktitle = {The Thirtieth AAAI Conference on Artificial Intelligence (AAAI)},
    Month = {February},
    Pdf = {http://www.robots.ox.ac.uk/~mobile/Papers/2016AAAI_ondruska.pdf},
    Title = {Deep Tracking: Seeing Beyond Seeing Using Recurrent Neural Networks},
    Year = {2016},
    github = {https://github.com/pondruska/DeepTracking}
    }