Meeting with Björn Mes about VR/AR development at Damen Schelde Naval Shipbuilding.
Meeting participants: Einar Lorentsen, Jostein Sætra Schefte and Björn Mes.
Agenda: Björn Mes works as technical specialist for VR/AR at Damen Schelde Naval Shipbuild-ing, a Dutch shipyard company providing maritime solutions through design, shipbuildShipbuild-ing, ship
repair and related services. Mes is a team leader for a small development team creating VR/AR applications.
Can you tell us more about the project the article (Virtual and Augmented Reality in Shipbuilding) was about?
Mes tells that the project was a proof-of-concept to find out if VR could be used for training at different scenarios. In the project they built a virtual training ground for specific technical equipment at a selected ship. The project proved successful, and a pilot project was started soon after. For the pilot project, Mes explains that his team created a replica of a 100-meter long ship in VR in order to test building bigger ship models. This was done by hand and was very tedious.
However, the result was a finished product which Damen today sells to its customers.
What is provided in the VR package Damen sell to costumers?
Mes tells that the VR models are used for training purposes. They provide maintenance training which can be specialized for the costumers for individual ships. When a ship is sold, a customer can choose from a wide variety of VR training packages in addition to requesting custom scenario packages.
How do you build the models of the vessels used in the virtual reality platform?
Mes explains that the models created in the article was done by hand, which was both time consuming and tedious. Mes’s team converted CAD models of the ship to 3D models supported by Unity. When doing this, the team had to clean up the textures and faces in order to make sure that it was possible for the technology at the time to run the application. Mes estimates that it took about 3500 hours to convert the models.
After completing the two projects, Mes wanted to reduce the time it took to convert the models.
Since Damen Naval are building the ships they provide VR packages for, Damen has all data which describes each vessel. Mes’s team realized that they did not need to manually extract 3D models from the CAD files, but could rather automatically generate the correct models based on the data already available to Damen. As an example, Mes tells that what once took 2-3 weeks to convert and clean up all the models of an engine room manually now takes about a minute. However, his team still have to mark and label objects manually in addition to some minor manual clean up of the vessel model.
Mes explains that his team generate the models automatically by accessing a database using Python, and find out if the object is a pipe, steel or equipment. If it is a pipe, they don’t export the geometry, but rather get the XML file and generate a model in Blender based on the this data. To convert difficult models they use Pixyz, which is a plugin for Unity, used to prepare and optimize large CAD models used in Unity. This is because they can’t automate big models easily, like the main engine, so they have to use other tools to help according to Mes. If some of the models are updated, Mes tells that Python keeps track of these changes and exports a new model.
Another advantage of having access to the metadata describing the ship is that you can auto-matically link systems that belong to eachother togheter. By doing this they can highlight whole systems, which from a didactic point-of-view very interesing according to Mes.
Are there any main challenges or difficulties to overcome when creating the models and using VR?
Mes tells that when they started using VR they had to connect the VR-headset to expensive gaming computers, and this limited the movement that was possible inside the model without using teleportation. This problem is now fixed by the Oculus Quest VR-headset which run wireless.
Another problem encountered was system confusion caused by light. Infrared light was used to track the player, and with reflection of light when there were many windows present caused problems.
This problem is also fixed as inside out tracking is used instead of outside in tracking.
Another problem is that it takes years to build and deliver vessels. As time goes by VR-headsets are updated, and the headset the application is meant for is not available anymore.
There are also problems related to the human factor tells Mes: “How can you make sure that everyone understands the interface? How can you make sure you don’t overfeed them with stuff?”.
Another problem is related to motion sickness in VR, in which Mes tells that you have to abide certain rules. They rules Mes states are: “never control the camera of the player, . . . and keep the frame rate as high as possible. Make your models and textures optimized so that you can have at least 90 frames per second. . . . and don’t make it an experience longer than 15-30 minutes.”
Mes has also experienced doubt related to the realism of VR. An example he comes up with is the absence of both sway that is present on a ship and the smell of oil that is not experienced in a virtual model.
Challenges Mes tells that Damen Naval have now is related to remote sensing, and how the latency effect this and what happens if the connection is lost. The safety of data strings sent from a ship to shore for a remote sensing session is also a challenge.
How do you optimize the models so it’s possible to run them smoothly?
Mes tells that they keep both the number of polygons low and the level of detail in the textures low to optimize performance. They also divide their ships into segments, use occlusion culling, and hide everything that is not visible. However, there are some edge cases that needs to be handled according to Mes. For instance, at a long hallway from the back of the ship to the front the tricks they use will not work. They also use game technology such as mipmaps, bake light before running the application to avoid live lighting, and keep the effects to the minimum. This is to avoid frame rate problems according to Mes.
Do you have any new VR projects you are working on?
Mes tells that they work on the scientific reasoning to use VR instead of real training. Another area they are interested in is adding simulations that behaves like a real system. This can be extended to help suppliers test their equipment in a VR environment.
What sort of display are you using to experience VR (head-mounted display, a single projection screen or multiple projection screen)?
Mes tells that they only use head-mounted displays. Before they used HTC Vive, but now they are using Oculus Quest 1 & 2. If a costumer needs a really specific training scenario that needs a lot of power, they have the option to use the Link cable with Oculus Quest, which connects the Quest headset to a computer in order to utilize its power.
Mes follows up by explaining that they don’t use projection screens. This is because if a multi-user experience is required, only one can be the master controlling what can be seen for the other users which become the slave. This caused a bad experience according to Mes. The systems also cost a lot of money compared to head-mounted displays.
Do you use VR in the shipbuilding phase?
Mes tells that they do, because VR is good to visualize something that hasn’t been built yet. So they use VR in the design phase as well. One time Mes remembers that they had a meeting in VR with a client where they went through the ship to discover if the dimensions of the ship was satisfactory. Mes tells that: “you can see stuff in VR that you can easily miss on the 2D plane.”.
Once they also had a VR session with engineers, and they discovered things they missed on the drawing because with VR you have more sense of the depth of size and your own location tells Mes.
Do you experience any increased interest from your costumers regarding VR?
Mes tells that they experience increased interest, especially by the northern European countries, and the NATO clients.
How realistic do you feel the models is compared to the real ship?
The shapes constructed are located at the exact same place and at the original size as the real ship, which cause realism according to Mes. Since the distances, sizes and positions of the buttons
are correct, you will recognize this in the real ship even though the ship doesn’t have the same lightning or textures. Mes tells that the textures aren’t like photographs yet, but they try to make the textures as real as possible. He explains that he once got onboard a vessel he had worked on in VR for an extended period of time, and that it was a weird feeling knowing the ins and outs of the ships layout even though he had never been on it before.