The remote control of heavy and risky construction machines such as excavators has always been a topic of interest. However, the lack of high-quality video streaming and low-latency control has posed a major obstacle to remote control. Fortunately, the rising trend 5G technology is making it more possible to remotely control construction machines by offering significant advantages. In this study, a pilot application for remote control of an excavator robot was carried out using VR goggles and control equipment over the 5G network. As a result of the pilot study, the operator was provided with an easy usage in a comfortable and safe environment.

The pilot study began with the necessary preparations for the active use of the excavator robot. Then, a virtual environment and video streaming screen were prepared for VR goggles usage in the Unity platform. Configuration settings were made for VR controllers to send task commands and a communication protocol was established between Unity and the excavator. On the excavator side, a camera connection was provided and a video streaming protocol was prepared. Both the excavator and Unity were separately connected to 5G networks. After preparing all infrastructure configurations and software, the pilot was tested over the Savonia 5G test network.
Preparation of the excavator robot

The excavator robot used in the application is a Raspberry Pi-based device. A control software has been written in Python language for the device to perform its movement tasks. The device uses a servo motor controlled hydraulic system and tank track system. When the control software is run on the excavator robot, it waits for commands from the client. The incoming commands are evaluated in the control software, transferred to the relevant terminal via an ADC converter and the excavator robot moves accordingly.

Preparation of the Unity VR environment

Unity is a cross-platform game engine developed by Unity Technologies used to develop video games and simulations. The aim with Unity is to show the 180-degree video stream from the excavator robot on the VR goggles worn by the operator. An MJPEG streamer was prepared for real-time transfer of the video stream. The video stream is transmitted from the excavator robot to the Unity platform using a dedicated IP address and port number via socket communication protocol. Libraries were prepared for the operator to control the excavator robot with VR controllers. Tasks were assigned to each button and trigger for both the right- and left-hand controllers. Commands created on the hand controllers are transferred from the Unity platform to the excavator robot via the socket communication protocol as with the video stream, but this time is the opposite direction.

Establishing camera connection and video data stream

A 180-degree camera is mounted with USB connection on the excavator robot to provide a comfortable view for the operator. The camera’s filed of view covers the entire front of the excavator robot, allowing the operator to obtain a broad and sufficient view during operations. An IP-based MJPEG streamer application has been implemented on the excavator robot to enable real-time transmission of high-resolution video data. Parameters such as video size and frame rate are configured in the MJPEG streamer. The video data is sent to the Unity platform via a socket communication protocol.

Setting up 5G connections

A network with wide bandwidth and high data speed is required to provide real-time high-resolution video stream from the excavator robot. Additionally, a network with must have very low latency for the device to respond instantly to the commands coming from the controllers. In this context, the Savonia 5G test network offers significant advantages over 4G LTE and Wi-Fi technologies, with its high speed and low latency.

The workstation to which the VR glasses are connected is connected to the Savonia test network with a 5G mmWave modem. Similarly, the excavator robot is connected to the Savonia test network with another 5G mmWave modem. Both modems connect to radio base stations in the private network using special SIM cards and dedicated IP addresses. As a result, a high-speed, low-latency and secure private connection is established between the device and the workstation.

Testing the Application on Savonia 5G Test Network

The application was tested on the Savonia 5G test network. The operator attempted to perform delicate tasks using the hand controller. It was observed that even in the case of sensitive tasks, real-time success was achieved thanks to the low latency provided by the 5G network. It was also noted that video data was more determined compared to a Wi-Fi connection. Positive feedback was received from the operator regarding ease of use.

Conclusion and Future Work

In this study, remote control of an excavator robot through a VR platform was achieved using 5G network technology. Thanks to the wide bandwidth, high speed and low latency of 5G technology, study has become much more reliable to remotely control the device. Real-time high-resolution video transmission and easy control using hand controllers enable operators to easily perform difficult tasks. Future work will focus on using 5G technology to remotely control different types of devices.

Video:
https://www.youtube.com/watch?v=im2QZc-_jdU

Project information:
5G:llä vauhtia digitalisaatioon (REACT-EU EAKR / Pohjois-Savon liitto, A77467)

Authors:

Osman Torunoglu, R&D Specialist, DigiCenter, Savonia University of Applied Sciences, osman.torunoglu@savonia.fi
Timo Lassila, Test Engineer, DigiCenter, Savonia University of Applied Sciences, timo.lassila@savonia.fi
Aki Happonen, Project Manager, DigiCenter, Savonia University of Applied Sciences, aki.happonen@savonia.fi
Unity3D VR environment developers:
Mahdi Belila, Savonia University of Applied Sciences, mahdi.belila@savonia.fi
Premton Canamusa, Savonia University of Applied Sciences, premton.canamusa@savonia.fi
Aihekategoriat (Topic categories):
Tekniikan ala,TKI, 5G (Technology, R&D, 5G)