Both a LiDAR scanner and a 360° camera will be used to create a realistic virtual model of a vessel. The LiDAR scans and 360° imagery will be taken by a crew member and sent to a surveyor that will explore the generated model in VR.
LiDAR Scanner
In order to acquire a 3D model that can be explored in VR, a scan performed by a LiDAR sensor is thought to give the best and most realistic results. There are a wide variety of LiDAR scanners available on the market today. Both handheld and stationary scanners are available with different resolutions. Since the LiDAR scanner used to gather material for the application will be used on ships which may be large in size and have a complex geometry, a handheld LiDAR scanner is thought to be the best option. However, many of the available handheld LiDAR scanners on the market today are expensive and difficult to operate.
In March 2020, Apple released its new iPad Pro series with a built in solid-state LiDAR as seen in Figure 3.1a. A LiDAR sensor was also included in the new iPhone 12 Pro series released on October 23, 2020. By this time, several new capabilities of the LiDAR sensor were added to the iOS operating system which demonstrates Apples commitment to the technology. The benefits of having a LiDAR in a phone or tablet includes better autofocus for cameras and more immersive AR.
LiDAR sensors are believed to be a mainstream feature of phones and tablets in the future, making 3D scanning of rooms and buildings easily accessible. It is likely that LiDAR sensors in future devices will be of even higher quality than the ones available in devices today. In addition, it is likely that algorithms used to process a LiDAR point cloud, generating its 3D shape and texturing it will continue improving, resulting in even more realistic results in the future. By gathering material for the application with a LiDAR sensor already available in current or future phones, the cost for Gard of making sure each vessel has access to a LiDAR scanner could potentially be zero, since it is likely that at least one crew member has a high end mobile telephone. In addition to it’s low cost, there already exists multiple apps on the App Store with a user friendly interface that allows users to scan and create 3D models and adjust scan parameters. This makes 3D scanning easily accessible. As a result, utilizing LiDAR sensors in current and future mobile devices to take scans used in the proposed application is regarded as the best option, as it gives a realistic result with a user friendly interface at a low cost.
360 camera
As the resolution of a LiDAR scan increases, the amounts of polygons in the generated 3D model increases accordingly. A model from a high resolution LiDAR scan will therefore require more computational power to render inside a VR headset and take up more space compared to a scan at a lower resolution. In order to make sure the proposed application runs smoothly on VR headsets, LiDAR scans will be done at a lower resolution than what is possible in the current sensors utilized.
To supplement the LiDAR scans and give surveyors the level of detail they require, a high resolution 360° camera will be used. Rendering a high resolution 360° image is computationally cheaper than increasing the resolution of the LiDAR scan. Newer 360° cameras offer high resolution images at a reasonable cost. By utilizing both LiDAR scans and 360° imagery, it is believed that one can achieve a better result as the features from both technologies will complement each other.
Capturing 360° images for the proposed application will be achieved with the GoPro Max as it is believed to be the best option. The camera can be seen in Figure 3.1b. The GoPro Max has two 180° lenses on both sides of the camera as seen in Figure 3.1b, and uses an algorithm to stitch the two photos together into a complete 360° degree view. Since the GoPro Max is an action camera, it has a robust build and is waterproof, which is essential properties for a camera being used by non-technical crew aboard a vessel. The GoPro Max has the ability to capture both photos and videos at 6K resolution, which is more than enough to capture detailed images. It is easily controlled via its touch screen or via an app on the phone, which makes it easier to use for first time users.
When capturing material for a survey, a crew member could simply put the camera on their head, turn on record, and walk through the vessel. The surveyor can then later extract the photos they need from the video. Alternatively, a crew member can walk through each room of the vessel and manually capture a photo with the camera. If the captured material does not adhere to the surveyor’s strict standards, he could request new material of desired objects or rooms until he is satisfied.
The GoPro max is priced between 400-500 USD, which is significantly cheaper than the expenses of flight tickets and accommodation required for performing a physical inspection. Assuming a crew member already has a high end mobile phone, the GoPro Max is Gard’s only required expense in order to conduct a remote survey. By sending the GoPro max to a shipowner and utilizing the LiDAR sensor on a crew members phone, the crew has all the equipment they need in order to gather material for a highly detailed realistic virtual model of the vessel. This material can then be sent directly to the surveyor that can inspect the material in VR.
(a) iPad Pro (2020) with LiDAR scanner to the right of the dual-camera system (Apple, 2020).
(b) GoPro Max 360° camera (GoPro, 2019).
Figure 3.1: The modeling tools used to gather material for the application.
VR headset
Once a surveyor has received sufficient material of satisfactory detail, he can explore it in VR. As discussed in subsection 2.1.1, there are multiple types of VR displays available. A head mounted display type is deemed as the best option as they are significantly cheaper and more accessible than stationary displays.
The proposed application will be developed using toolkits that allow for cross-platform VR devel-opment. Even though the proposed application will be developed to work on other head mounted VR headsets, the preferred head mounted display for testing and developing the application will be the Oculus Quest 2 seen in Figure 3.2. The Oculus Quest 2 offers a fast and responsive VR experience at a reasonable price. In addition, as discussed in subsection 2.1.1 and by Bjorn Mes (section B.3), the Oculus Quest 2 is capable of running applications both internally and externally.
This means that the Quest 2 is able to run without being connected to a an external computer. If increased computational power is desired, the Quest 2 can be connected to an external computer via a link cable and run applications from the computer.
Being able to choose whether to run an application internally or externally gives surveyors great flexibility. After a crew member has captured a high resolution LiDAR scan of a vessel, it is possible to run a simplification algorithm on the captured 3D model inside a scanner app to reduce its polygon count. When the crew member send both the original and the simplified 3D model to the surveyor, the surveyor can choose which model he wants to explore based on his current hardware situation. If the surveyor does not have a powerful computer next to him, he can explore the model with a reduced polygon count internally in his headset and still have a high frame rate. However, if he has a powerful computer available, he can take advantage of its increased computational resources and explore the high resolution model running the application externally from the computer.
Figure 3.2: The Oculus Quest 2 (Oculus, 2020).