R&D project HOBA
1. Introduction
The R&D project "Homogeneous soils assistant for the automatic, construction site-specific recording of soil classes according to the new VOB 2016 HOBA”, shortly HOBA, deals with the development of a system for an automatic classification, detection & segmentation and a georeferenced voxel-based 3D-volume model generation for excavation site specific soil types according to the new VOB 2016.
HOBA is financed as a so-called ZIM (Central Innovation Program for SMEs) research and development project by the Federal Ministry for Economic Affairs and Energy (BMWI) up to 03/2023. HOBA is located at the Institute for Applied Research (IAF) of the Center for Applied Research (CAR) at Karlsruhe University of Applied Sciences (HKA). Research and development are carried out in the GNSS & Navigation Laboratory (goca.info/Labor.GNSS.und.Navigation ) in collaboration with the main industry partner MTS Schrode AG (www.mts-online.de) and their partner VEMCON GmbH (www.vemcon.de ).
Figure 1: Excavator with distributed MTS sensors and HKA HOBA-Box located in the area of the excavator bucket
The aim of the R&D at the HKA is the development of the hardware and software of a compact sensor and a computing system unit, mounted on the excavator with data interfaces to the excavator IT - hereinafter referred to as "HKA HOBA-Box".
The hardware and software development of the HKA HOBA-Box is an innovative contribution to the BIM-compliant digital real-time documentation of excavation work. Here, the HKA HOBA-Box enables a multi-sensory 3D geo-referencing of the excavation in the ETRF89/ITRF in connection with the sensor-based acquisition (GNSS/MEMS/RGB/ToF-3D-camera Optics), and so a calculation of a so-called "voxel"-based 3D model of the excavation volume. This means the box will allow the classification of the soil types on the site using image-based AI/ML algorithms and finally do the re-calculation of the classified and georeferenced 2D images into the geo-referenced 3D voxel model according to the soil types.
The complete geo-referencing steps of the box is based on algorithmic fusion and SLAM of all internal sensor data of the HKA HOBA-Box (IMU, magnetometer, barometer, inclinometer, GNSS, ToF-3D-camera and RGB-camera) in the general NAVKA multisensors-multiplatform leverarm design. All sensor data will contribute to the calculation of a Bayesian sensor-fusion in favour of the navigation state vector
In case of SLAM (Simultaneous Localization and Mapping) in favour of a fusion and a state vector y(t)SLAM=(y(t),m(t)). The extension of y(t) in case of SLAM is based on the optical sensor-data of the ToF and digital RGB-camera, and parameter space of the 3D map m(t).
In the self-sufficient box variant 1 of the HKA HOBA-Box (Fig. 1, Fig. 2) uses only the data from the HKA HOBA-Box for the sensor fusion in respect to the estimation of y(t), or the SLAM parameters y(t)SLAM=(y(t),m(t)), respectively.
In the case of box variant 2, y(t) and vector y(t)SLAM=(y(t),m(t)) are calculated on the HKA HOBA-Box again from all sensors, except the GNSS. The reason is, the unfavorable placement of GNSS at the location of the box in the vicinity of the bucket with regard to signal shading, multipath and cycle slips. Instead, the sensor fusion and SLAM on the box variant 2 are making use in addition of the MTS navigation part solution y(t)'=(xe ye ze )T provided via the local machine server (see local machine server, fig. 2) at the localization of the body (b) and box origin, respectively.
The HKA HOBA-Box - in both variants - enables a multi-sensory 3D geo-referencing and voxel-based classified 3D model of the excavation volume in the ETRF89 / ITRF based on (GNSS/MEMS/RGB/ToF-3D-camera Optics) sensor data. The classification of soil types on the site is based on image-related AI/ML algorithms, and finally the re-calculation of the classified and geo-referenced 2D images into the above-mentioned geo-referenced 3D voxel model according to the soil types.
2. Hardware Developments
The concept of the hardware design as shown in fig. 2. As central processor unit an NVIDIA Jetson TX2 with a 256-core NVIDIA Pascal GPU, a hex-core ARMv8 64-bit CPU complex and an 8 GB LPDDR4 memory with 128-bit interface is used on the box. The CPU combines one dual-core NVIDIA Denver and 2 processors with a quad-core Cortex-A57 arm. The HKA HOBA-Box internal sensors include ZED F9 GNSS, compact ICM 20948 sensors (3-axis gyroscope, 3-axis magnetometer and 3-axis accelerometers), and MS5611 barometer and an SCL3300 inclinometer (tilt meter). This box is the above box-variant 1, while variant 2 uses a part of the navigation state y(t)’ - namely position the position y(t)’= (xe ye ze) - calculated by MTS from the two GNSS and tiltmeters on the excavator machine (fig. 1) by the use of the Denavit-Hartenberg transformation.
The used backup drive is a WD Red SA500 NAS SATA SSD 1 TB and is used for data backup via defined threshold values and/or a defined backup rate (e.g. hourly).
Figure 2: Design concept of the HKA HOBA-Box and the data communication on the Excavation machine
The LUCID Helios-2 TOF with integrated RBG IP67 kit (Triton 3.2MP) is used as the optical component of the HKA HOBA-Box. The additional digital camera of the LU-CID Helios-2 TOF could is used mainly for the ML/AI-based classification, as well as for texturing the TOF generated point clouds.
The prototype box (fig.3) is set up as full system for the running navigation and SLAM, image processing, voxel generation and georeferencing algorithms and software developments, while the final smart box is under development as well.
Figure 3: Prototype box for system algorithms and software development 1.) ToF, 2.) Back up drive, 3.) Additional Giga-Ethernet ports, 4.) Digital camera MIPI CSI, 5.) Additional pins for CAN and other functions.
3. Algorithms and Software Developments
The previous hardware design is used as a base for the algorithms and software implementations as described in fig. 4.
All the calculations are processed centrally on the HKA HOBA-Box, i.e. the NVIDIA Jetson TX2 computer (NVIDIA Pascal Architecture GPU, 8 GB L 128 bit DDR4 memory, 32 GB eMMC 5.1 flash memory and 1.0 TB external memory) there. The operating system of the HKA HOBA-Box is Linux 18.04 LTS "Bionic Beaver" with ROS Distribution (Robot Operating System-Melodic). The system is already prepared for deep learning based on Python and C/C++ with a lot of necessary dependencies installed.
Initial experimental tests using TF1.x and PyTorch have successfully been carried out on image classification, detection and segmentation using transfer learning with different pre-trained models e.g., ReNet, FCN- ReNet, SSD-Mobilenet, Inception V3, etc.
