Smart window—augmented reality for tomorrow's ship management

The precisely transmitted image of the quay wall on the ship's window of the steering bridge, provided with some additional information, for example on current conditions and mooring possibilities—even in fog, darkness and poor visibility—this is what the ship navigation of tomorrow could look like.

With the joint project "Smart Window," Fraunhofer ISIT has worked together with partners from industry and the Fraunhofer Institute for Computer Graphics Research (IGD) on an intelligent solution for implementing an augmented reality display on a ship's bridge in order to process, filter and display all relevant data for ship operation in the future.

Embedding the AR display in a window of the ship's bridge offers the possibility of displaying a variety of information directly in the skipper's field of vision, thus facilitating the skipper's work.

Fraunhofer ISIT has developed a scanning full-color laser projection system as the central element for the novel AR display. The core of the system is a silicon micromirror. It is suspended on two perpendicular axes. The mirror can oscillate about these axes, excited by electrostatic forces. When laser light is cast onto the mirror, such a mirror can be used to project any image onto the ship's window that is sufficiently large and well illuminated for the application. To ensure that the mirror achieves large deflection angles and does not become dirty, it is encapsulated with a glass cover at low internal pressure at the wafer level.

The project as a whole is very challenging. An augmented reality view is created by the complex interaction of data preparation, sensor technology and visualization, which is usually created specifically for one viewing angle. In order to feed information into the user's field of view in such a way that it specifically enriches reality, the digital model of the world must be brought into congruence with reality.

For example, global data, such as position data from the electronic nautical charting system, and local data from the ship's own sensors, such as radar, must be fused and rendered relative to the viewer. In addition to enormous computing power, this requires information about the user's field of vision in particular, so that the superimposed data can be clearly assigned to real objects from the user's point of view. If the user moves, this must be tracked in real time and the virtual image adjusted accordingly.

This tracking function is performed by a 3D camera, which must function robustly and accurately even in the low light conditions on a ship's bridge at night.

Another important component for AR projections on the windows of the steering bridge are suitable glass laminates for the windows. These consist of standard-compliant individual glasses bonded with films to form a laminate, which must meet the stringent requirements for suitability in maritime environments.

The Smart Window project has great significance for the increasing demands on the current transport and supply infrastructure in the face of increasing globalization. Shipping plays a special role here. This is because while truck drivers and train drivers are tied to the infrastructure of the road and rail network, and pilots can rely on electronic monitoring and control worldwide, ships are often still left to their own devices in many places. This requires a high level of decision-making authority and forward-looking data-based planning on the part of ship operators.

"In particular, up-to-date comprehensive information on navigation, weather and local features such as crossing ships is available in real time with Smart Window, as is information on the technical status of the ship and the functionality of many subsystems. Smart Window thus offers good approaches to further optimize safety in maritime traffic," explains the head of the project at Fraunhofer ISIT, Dr. Shanshan Gu-Stoppel.

In the past few months, the scientists at Fraunhofer ISIT have once again significantly improved the projection system and the necessary control electronics with regard to image stability, contrast, stability and refresh rate. They will be demonstrating their system at the Hannover Messe from April 17 to April 21, 2023, at the Fraunhofer-Gesellschaft's booth A12 in Hall 16.