Algorithms in the Arctic: Removing bad weather from images to make Arctic shipping safer

Arctic shipping traffic is on the increase. One day, these ships will be autonomous. New technology that can remove rain, snow and fog from the images produced by the ship's cameras and sensors will increase safety in extreme conditions.

Imagine an autonomous ship sailing through one of the world's most extreme ocean areas. Sea ice is everywhere. Fog, snow or rain make visibility extremely poor. Just like ship captains see through their eyes, autonomous navigation algorithms perceive the world through sensors, and bad weather is just as impenetrable for sensors as it is for sea captains.

Getting rid of poor visibility

With the rise of Arctic shipping, something that can remove the bad weather from the images so the algorithms can see the surroundings as if it were a clear, sunny day could be extremely useful. Now, Ph.D. candidate Nabil Panchi at NTNU's Department of Marine Technology has developed an algorithm that can do just that.

"We have put in place a new piece of the big puzzle for better modeling of sea ice," Panchi said.

Current AI algorithms work well on clear images, but they struggle when images become blurry or degraded due to bad weather.

Panchi, who is also a naval architect, has used thousands of images from the Arctic to train the new algorithm so it filters out visual impediments such as rain, snow, and fog, as well as water droplets on the lenses of the cameras that many vessels are equipped with.

Panchi is affiliated with the DigitalSeaIce project, which is focused on multi-scale integration and digitalization of Arctic sea ice observations and prediction models. The main objective is to build a digital infrastructure that integrates regional sea ice forecasting models and local ice-related models with shipboard and satellite-based Arctic sea ice and environmental observations.

Understanding the environment via images

"Our work is about understanding the Arctic environment through the use of images. We are creating algorithms that work in all weather conditions" says Panchi.

His research is based on thousands of images taken on a voyage with the research ship Kronprins Haakon in the Arctic during the summer of 2023.

In collaboration with his academic supervisor, Associate Professor Ekaterina Kim, he published the article "Deep Learning Strategies for Analysis of Weather-Degraded Optical Sea Ice Images" in IEEE Sensors.

Panchi and Kim are introducing two ways of helping ships travel more safely in bad weather in the Arctic, by "removing" the weather from images. One uses artificial intelligence to clean up the images, so that existing algorithms work as they should. A slightly more efficient way is to develop new algorithms that work during bad weather.

"Both strategies allow us to understand the Arctic in all weather conditions," Nabil says.

Cleaned images already in use in cities

Algorithms that can remove weather from images have been in use for a long time, but primarily in urban areas. They are used to develop autonomous cars, and in security and camera surveillance.

Current algorithms that analyze sea ice are largely based on images taken from ships in good weather conditions. The problem is that images from the Arctic are often unclear due to the fog, rain, and snow that are common weather conditions in these waters. These types of images are poor material for the existing algorithms that are designed to understand the Arctic environment.

The algorithms also need to be trained to analyze the type of ice surrounding the ship, so they can indicate where it is safe to break through the ice, and which areas the ship should avoid.

The first open-access dataset of sea ice images

In order to remove fog and raindrops, algorithms must be trained to clean up weather-affected sea ice images.

"This area of research had largely been ignored so far. The problem has been limited access to clear images from the Arctic—until now. We hope that our new open-access dataset helps in future development of weather resilient technology," Panchi says.

Panchi's supervisor Ekaterina Kim has worked extensively in the Arctic, and in recent years she has been exploring how AI can be adopted to solve some of the challenges that exist in polar regions.

The two NTNU researchers have now made the SeaIceWeather dataset publicly available online. It contains thousands of images and is the first open-access data set for sea ice.

Facilitating safer voyages

"There are very few open-access datasets of this type. A huge amount of work goes into making them. We hope they are used as much as possible," says Panchi.

Each picture comes in two versions: a 'clean' version with a clear view, and an unclear one due to weather conditions. NTNU researchers hope that the SeaIceWeather dataset will be used by more people and that it inspires them to collect these types of images.

Many of the users are researchers who are working on sea ice and navigation models, or dynamic positioning. These systems must work in all weather conditions, and the more images the algorithms are given to learn from, the more accurate the monitoring, ice warnings, and navigation will become—something which is very much in demand.

More ships—and inexperienced captains

Global warming is causing sea ice to melt, increasing the amount of Arctic shipping. More and more shipping companies are choosing these new routes that have now become ice-free. Between 2013 and 2019, ship traffic in the Arctic increased by 25 percent.

"It takes a lot of experience to navigate safely through sea ice. There are probably more ships in polar waters now than there are experienced sea ice captains. The system we have built can provide better assistance for people maneuvering the ships," says Panchi.

Arctic sea ice has become thinner, cracks more easily, and can make massive ice ridges or hummocks. From the bridge of a ship, only one meter of ice might be visible sticking up above the surface, but not the 4–5 meters hidden below. The likelihood of dents and hull damage is high, and not all ships are built to break through ice.

At the same time, autonomous shipping holds the potential to revolutionize the shipping industry—making it more efficient and safer. According to Fortune Business Insights, the global autonomous ships market size is projected to grow from $6.11 billion in 2024 to $12.25 billion by 2032.

"We expect more autonomous technology on ships navigating through ice, and current systems need to be trustworthy in the extreme Arctic environment," says Kim.

30 days of data capture

Panchi has trained the algorithms on two image datasets: one collected during the GoNorth voyage on Kronprins Haakon, the other obtained from online image databases.

He mounted two cameras on one side of the ship, with one camera directly under the other one. The upper camera had a clear view, while they had mounted a transparent screen in front of the lower camera, which was sprayed with water to simulate raindrops on the lens.

In the observation room on the ninth deck, Panchi's computer continuously downloaded the images of sea ice. For 30 days, he collected thousands of pairs of images, each of which consisted of one clear image and one covered with artificial rain.

Training algorithms

In total, the datasets consist of over 4,600 clear images, most of them from the research voyage. Using algorithms, they created seven weather variants for each clear image: small, medium and large snowflakes, rainy weather, fog, and real and simulated raindrops on the camera lens.

Based on these variants, they then created the two datasets. One of them presents images that indicate what kind of ice is located around the ship. The other dataset divides the ice into different categories, such as ice floes, pancake ice, ice slush, drift ice, etc.

Three different image-cleaning algorithms were trained on the datasets, and when the researchers compared the results with the clear images, they could easily tell which algorithms were most accurate in relation to the different types of weather.

Only daylight and three types of weather

The method is limited in that all the images are taken in broad daylight and involve only three types of weather conditions. Panchi points out that since the Arctic is in darkness from September to March, similar images should ideally be collected during the winter.

However, it is also fully possible to use augmented reality and create an artificial winter or night-time version of the existing images.

"So far, it's mainly researchers who can use what we have done, but we hope that more people will use it in the future. There are many factors affecting when this will actually happen; it may take up to five years before the models can be used commercially. They must then be of a quality that make them a fully reliable assistant for ship management," Nabil said.

Reduced emissions

The largest ships use enormous amounts of fuel. They sometimes have to sail back and forth into the ice in order to break through, which requires a lot of energy.

"If you fully understand the conditions surrounding the boat, you or AI can plan the route and save time, effort and therefore emissions. It will also make shipping safer. There are a growing number of tankers carrying liquefied natural gas and other cargo sailing through the Arctic. So far, there haven't been any oil spill incidents, but if one were to occur, it would have serious consequences," says Panchi.

A lot of unused image data

Monitoring polar waters is also important with regard to climate change. Many ships have cameras and sensors that monitor their course. There are lots of ships producing images, but hardly any of these images are available online. According to Panchi, most of the images end up in maritime data archives, and—apart from a few insurance cases—are never used.

"We see great opportunities in extracting useful data from these images. One of our goals is to develop algorithms that can be improved in real time, on site. Improving how we monitor Arctic waters will benefit society. It will provide a better basis for forming policies, and for sustainable and safe use of Arctic waters," Panchi said.