Taking control of unmanned vessels at sea

Norway is taking the helm to develop technology that will be used to send unmanned vessels to sea on assignments without a crew on board.

The researchers face some daunting challenges, such as determining what kind of control systems are needed to create safe traffic at sea for autonomous vessels and figuring out how a driverless ferry should be designed so that passengers will trust that it is safe to board.

An interdisciplinary research team at NTNU is now taking a deep dive into these work areas.

Laboratory by the sea

The research team, which works on design and control systems for driverless vehicles, is now building a completely new lab in the middle of a maritime environment, close to the sea where they can develop and test new technology exactly where it will be used. The lab’s new name is the NTNU Shore Control Centre.

The premises will soon be ready for occupancy. Photo: Idun Haugan / NTNU SHOW MORE

Nyhavna in Trondheim is an area filled with old and new industrial companies. Next door to the Dora WWII submarine bunker is the Trondheim Maritime Centre,  where researchers, designers and technologists are in the process of building the technical infrastructure in the new lab.

Over 600 square metres of offices and meeting rooms are ready to receive students, fellows and researchers, but the coronavirus has put the move on hold. In the meantime, work is being done to furnish the very heart of the research environment: a technical research lab that will be more like a captain’s wheelhouse – but one that is located on land.

Human factor in the technology

“Autonomous transport is a technology that we’re developing for export,” says Ole Andreas Alsos, head of the Department of Design. He predicts that this industry is going to be big and  something we’ll live off in the future.

The work he leads is part of a newly started SFI – the Norwegian acronym for the Research Council of Norway’s Centre for Research-based Innovation programmes. These centres consist of many partners from universities and businesses, and are intended to enhance the strength and speed at which new disciplines and technology can be realized.

The goal of SFI AutoShip is to ensure that Norwegian players take a leading role in the development of autonomous ships, nationally and internationally. NTNU’s Department of Engineering Cybernetics heads SFI AutoShip under Professor Mary-Ann Lundteigen.

You can more about the entire initiative in the SFI Autoship fact box below.

The Department of Design has been assigned to manage what is known as a “work package” in the SFI for the section that is defined as Human factors.

“We’ll ensure that the human factor has a place in this artificial intelligence and computer world,” says Alsos.

World’s first test area for driverless vessels

Turning the clock back four years, the Trondheim fjord was inaugurated as the world’s first test area for driverless vessels. This inspired Egil Eide, an associate professor in NTNU’s Department of Electronic Systems, and a team of students to build Norway’s first driverless ferry. It was named milliAmpère.

• You might also like: Test site opens for unmanned vessels

The milliAmpère ferry was designed and built to test how to carry passengers across the Channel in Trondheim, a distance of about 100 metres. In earlier times, a ferryman rowed passengers across the Channel. The milliAmpère team based its work on this old tradition when they developed a completely new type of vessel.

The milliAmpère is a pilot project that has been undergoing testing for the past three years. It is 5 metres long and 2.8 metres wide – so not very big.

Eide has been collaborating with students and PhD candidates.  He is using his experiences from milliAmpère 1 and is now in the process of building milliAmpère 2, which will soon be completed. It is twice as long and twice as wide as its predecessor, contains far more technical equipment and has a distinctly different design.

The actual hull of aluminium was made at a shipyard on the island of Frøya, and now Eide is installing the technical equipment with the help of fitters and PhD candidates. About 20 master’s students and 3-4 PhD candidates have contributed in the development of the technology and algorithms for the driverless ferries.

This is what the new ferry that will go into traffic across the Channel in Trondheim will look like. Design: Petter Mustvedt, Department of Design SHOW MORE

Filling the ferry with technology

Eide says that the research team gained a lot of experience from the first ferry version. In version two, the hull is being filled with lots of technology.

Associate Professor Egil Eide leads the work on generation two of the driverless ferry. Photo: Idun Haugan / NTNU SHOW MORE

“One of the things we learned from the first version is that we need a lot of cabinet space for technical gadgets. Now we’re filling the space under the deck with battery boxes, the operating system, chargers and several powerful computers.

Sensors, cameras, lasers and radar are being mounted around the deck.

“The experiences have taught us how sensors and cameras should be placed so that passengers don’t block them,” Eide says.

Design insights are important, too.

“The railings and bench surfaces are made of wood in the new version to give a warmer feel. It’ll also be quite spacious so that passengers feel secure and not crowded. The ferry has room for 20 passengers, but we’ll operate with a maximum of 12 passengers on board,” says Eide.

Ferry model: Petter Mustvedt, Department of Design SHOW MORE

“milliAmpère 1 was an engineering feat. In milliAmpère 2, we’ve brought design to the fore so that it’ll feel safer for passengers,” says Alsos.

Design is one of several elements being addressed, and another one is how to design the ferry so that kayakers and leisure boats recognize what manoeuvres it’s planning.

“The ferry’s still a bit uncommunicative, so we have to develop a way for it to signal to the surroundings what it plans to do in the next seconds and minutes. Being able to discern the ferry’s intention – like whether it’s about to turn right or left – is important,” says Alsos.

Numerous professional groups have contributed to the ferry’s financing, development and testing, which has involved broad interdisciplinary work. The vessels are defined as interdisciplinary research platforms.

• You might also like: Driverless ferries to replace footbridges

Students build model ships

milliAmpère 2 will be a test vessel and comprise an essential element in the work at the new lab.

But more model ships will also be built in the future by students connected to the lab. A project modelled on the Revolve NTNU model is being developed, where students collaborate to build driverless vehicles in order to enter their creations in the international Njord Challenge competition.

Just outside the panoramic windows of the lab is the harbour pool. Here, a long floating jetty has been installed, which becomes a platform for testing driverless model ships. The distances between the production facilities, control room and test area are conveniently short, in other words.

The harbor basin at Nyhavna, right next to the new lab, will be a test area for student-built model ships. Photo: Idun Haugan / NTNU SHOW MORE

The captain goes ashore

Human beings are still needed to monitor and ensure safety for unmanned vessels. The new kind of captain sits in a control room on land and oversees that everything goes as planned and intervenes if problems arise.

“What we’re working on is the control room for the operators who’ll be monitoring the vessels. That’s the expertise we’re developing in this new lab,” says Alsos.

The main goals are to:

• Design good control rooms that allow the operator to effectively monitor the vessels.
• Create good sensors and good algorithms for the vessels to avoid and steer away from obstacles.
• Develop dynamic artificial intelligence that manages to make the right decisions.
• Provide operators with a solid understanding of potential situations so that they can quickly address any problems that arise.

Important elements in the control room are a large array of displays where operators can follow unmanned vessels being tested out at sea. The vessels are equipped with a large set of sensors to communicate with the control room and with the environment where they’re located.

“We’ll use the operators to figure out how to design the displays and to determine what data the operators need,” Alsos says.

“We’ll also measure their reactions using advanced eye tracking that captures how the operators are reading the information on the screen. Biometric sensors will measure the operators’ stress levels based on their heart rate and breathing rate.

From this data the researchers can find out exactly how much time the operators need to gain an overview and the situational awareness necessary to make the right decisions.

“The operator also has to be able to talk to the passengers on the ferry when needed, and be able to remotely take control of the ferry and steer it safely ashore,” says Alsos.

The drawing shows what the control room will contain:

Illustration / scale model: Thomas Kaland, Department of Design SHOW MORE

Starting as a computer game

Erik Veitch, a PhD candidate at NTNU’s Department of Design, is a key player in the research and testing taking place in the control room.

“We’re starting with a simulation. It’ll be like a computer game world for testing different scenarios, like accidents, to see what kind of information future operators in the control room will need in a real situations,” he says.

As the operators run the simulation, the researchers concurrently sit in another room and monitor the operators. From here they measure their reactions and can activate various parameters to test for optimal data setup, the best screen layout, and what information the operators need to react quickly and safely.

“By simply starting as a video game, everything happens in a safe environment where things can crash and the autonomous vessels can run over kayakers and so on. At the same time, we can follow the operators and measure how they respond,” Alsos says.

Then off to sea

The next step will be to place an operator on board milliAmpère 2, which will run shuttle traffic across the Channel in Trondheim.

“Once we see that things are going well, the operator will monitor the crossings from land. Once that works, the operator will sit here in the control room and intervene if something happens. In the long run, an operator might be able to manage several ships at the same time. And in the future, the operator might not be needed at all. But if and when we might reach that point is hard to say,” says Alsos.

A human presence will probably be necessary for the foreseeable future.

“It’s a paradox that humans are still the most important component in the entire technological system with autonomous vessels. The more technology we develop, the more we need people to manage, control and have a systemic understanding of it,” says Veitch.

It must be felt safe to board. Illustration: Petter Mustvedt, Department of Design SHOW MORE

An alternative to building expensive bridges

But why do we want to replace people with technology?

“A digital captain only extremely rarely makes mistakes. Quite simply, human errors are eliminated,” says Alsos.

“Another advantage is that the captain and crew can work safely from land and avoid being away from family and out at sea for long periods,” he adds.

Another factor is cost. One project that illustrates this is the ferry-free E39 coastal highway route. The plan for the major ferry-free E39 road project is to be able to drive from Kristiansand to Trondheim without having to take a car ferry anywhere along the way. This requires bridges to be built over multiple western fjords. And bridge building is expensive.

“Driverless ferries cost a fraction of what a bridge costs. The E39 will cost huge sums of money, while a fleet of autonomous ferries that are there when you need to cross the fjord would be much, much cheaper,” Alsos says.

Alsos doesn’t want to speculate whether the technology will be far enough along so that driverless ferries could become an alternative to bridge construction on the E39. But autonomous vehicles will eventually become an alternative to expensive bridge-building projects. And this is one of the focus areas that SFI AutoShip is involved in developing.

Interdisciplinary collaboration

“What’s completely unique about SFI AutoShip is the interdisciplinary collaboration,” Alsos says.

The centre consists of a total of 23 partners from academia, business, government and research institutes. NTNU participants include the Department of Design, the Department of Electronic Systems, the Department of Marine Engineering, the Department of Information Security and Communication Technology and the Department of ICT and Natural Sciences in Ålesund. NTNU’s Department of Engineering Cybernetics heads the SFI centre.

“My idea in the long run is to also involve sociologists and psychologists. They can contribute with analyses of how passengers experience boarding a self-propelled ferry without a crew and what steps can be taken so they trust the technology and feel safe on board.