Teleoperation enhancement for improved control of unmanned ground vehicles through video transformation and deep learning based future frame prediction
Date of Award
2023
Document Type
Thesis
Publisher
Edith Cowan University
Degree Name
Doctor of Philosophy
School
School of Engineering
First Supervisor
Alexander Rassau
Second Supervisor
Douglas Chai
Third Supervisor
Syed Mohammed Shamsul Islam
Abstract
In this era of the fourth industrial revolution, the advancement of computer systems, artificial intelligence, computer vision, and automation are playing increasingly significant roles. The rapid levels of advancement in these technologies are making robotics an integral part of many aspects of human venture. More specifically, for applications such as item delivery, telesurgery, industrial production, search and rescue, mining, underwater, aerial and space exploration, and defence, robotic systems are becoming increasingly capable. Although the expectation is that such robotic systems will become increasingly autonomous, building autonomous systems that are capable of performing such complex tasks without supervision will require significant further research. Until this is achieved, robotic teleoperation will be an important component of these applications. Effective teleoperation of remote systems is, however, highly challenging with considerable opportunity for development and improvement. In response to this need, the aim of this research was to explore novel methods for the enhancement of robotic teleportation of unmanned ground robotic vehicles operating in complex and unstructured environments.
Ground vehicle teleoperators face a number of challenges including limited
situational awareness, deficiencies in the communication medium, and, latency or delay in the communication loop. Latency in particular can significantly impact
teleoperator performance through overcorrection-induced oscillations, giving rise
to increased task completion time, increased cognitive load, and risk of damage to robotic vehicles and the remote environments they operate in. To identify solutions to these problems, this thesis (a) explores existing teleoperation methods and enhancement techniques, (b) identifies both conventional and artificial intelligence (AI) based techniques that have the potential to enhance teleoperation through latency compensation, and (c) explores new methods and techniques for evaluation of teleoperation enhancement approaches.
A teleoperation simulator that can emulate ground vehicle robotic teleoperation
with adjustable latency has been developed. Five datasets have been generated
using the simulator that can be used for AI based teleoperation enhancement
research. A video transformation-based predictive technique has been proposed
and its effectiveness validated via a human operator-involved survey. A novel
conditional generative adversarial neural network based future video prediction
technique has also been developed that uses optical flow informed image-to-image translation for long future frame prediction for effective latency compensation. A perpetual loss function has been applied to this network for improved structural integrity preservation. An evaluation of the new techniques has been carried out using the PSNR, SSIM, and Multi-SSIM image comparison metrics. Statistical analyses of the results have confirmed the promise of the developed techniques for latency compensation based teleoperation enhancement.
This project is the first to have applied deep learning based future frame prediction approaches to the problem of latency compensation for teleoperation enhancement. The effectiveness of the proposed prediction based latency compensation approach has been demonstrated through human operator testing. Highly promising results have also been achieved for image-to-image translation based future frame prediction for further enhancement of predictive feeds. Additional experimentation on the architecture of the generative adversarial network, and development of a more sophisticated generator loss function is needed to further improve the image quality and help predict frames deeper into the future time horizon. Future integration of the AI-based model into a real-life robotic teleoperation system will offer further insights into the techniques proposed.
Access Note
Access to this thesis is embargoed until 12th April 2024.
Recommended Citation
Moniruzzaman, M. (2023). Teleoperation enhancement for improved control of unmanned ground vehicles through video transformation and deep learning based future frame prediction. Edith Cowan University. Retrieved from https://ro.ecu.edu.au/theses/2644