There is an increased need for unmanned and autonomous systems across the military, especially the U.S. Navy and Marine Corps. While unmanned surface vessels (USVs) can reduce the cost and human operators required to perform certain military missions, current technologies to perceive and understand their surroundings are not always reliable.  

Currently, there is not a reliable perception system for USVs for Navy and Marine Corps missions in a littoral or nearshore environment. But Mechanical Engineering Associate Professor Yi Wang is collaborating with several College of Engineering and Computing colleagues and Integer Technologies, a Columbia-based applied research and product development company, to create a new technology suited for harsh maritime environments.

Wang and his team received a $4.75 million, three-year contract in November from the Office of Naval Research for, “Reliable Perception for Unmanned Maritime Systems.” Wang is principal investigator along with three co-PIs from the college and Dr. Josh Knight of Integer Technologies.

“The long-term goal is to enable automated maritime perception systems for USVs that are better than human performance and provide reliable situational awareness of natural and man-made objects that influence navigation,” Wang says.

Due to complex and rapidly changing environmental conditions, current USV technology requires considerable operator oversight. For navigation, including avoiding obstacles, USVs also use sensors consisting of radar, GPS, and electro-optical/infrared (EO/IR) camera systems. Most of these sensors have been designed and adapted from other applications which can cause major accuracy and reliability errors. The radar systems also emit an electromagnetic signature, increasing the probability of detection in certain contested environments. Wang and his research team plan to develop AI-enabled passive EO/IR camera systems for improved perception and situational awareness that can perform better than a human operator.

This is only the starting point. We are confident that this project will produce a knowledge base and technology demonstrations that will make advances critical to providing the state-of-the art capability.

-Yi Wang, Mechanical Engineering

The research will be divided into three stages: physics-based analysis, perception system development, and technology transition. The physics-based analysis will evaluate the performance of various sensor configurations as a function of operational conditions, such as the strength of the signal emitted from target objects and weather conditions. This testing phase will be performed in nearby Lake Murray and in the waters off the coast of Charleston.

“We will do physics-based analysis and customized sensor configurations to understand and investigate the special characteristics and quality of images taken by the sensors and their dependence on different environmental conditions. This will give us a fundamental understanding of the challenges for the perception system,” Wang says.

The second stage will focus on fully developing and evaluating the customized perception system. Finally, the new technology will be integrated and demonstrated on a USV in relevant maritime operating environments. The goal is to improve the detection and recognition of other vessels and objects around the USV before passing real time situational awareness information to an autonomy system to improve the safety and effectiveness of navigation and other vessel behaviors.

“Perception performance in new scenarios will be more robust if emphasis is placed on understanding low-level features in images that are more likely to transfer. We also want to explore some multi-band, multi-spectral imaging technology to further increase accuracy, especially at night,” Wang says.

Integer Technologies CEO Duke Hartman hopes the technology developed by the joint research team will perform as good or better than a well-trained ship captain.  

“Right now, the radar has to be tuned to environmental conditions, and it can miss things if not tuned properly. EO/IR systems are not typically used for navigation and other autonomy behaviors. Our system will take the place of a human with good vision who understands the ocean environment,” Hartman says. “We're trying to achieve human-like perception or better to detect obstacles and threats, and make judgements in all range of environmental conditions, such as at night, in fog or rain.”

The co-PIs for the project are electrical engineering professors David Matolak and Bin Zhang, and computer science and engineering professor Yan Tong. Matolak will help develop fundamental physics-based analysis of the radar in littoral environment. Zhang will support Guidance, Navigation and Control algorithms in the new platform for data collection, while Tong will work on high-dynamic range imaging, computer vision, and sensor configuration.

Integer Technologies will provide sensor selection, prototyping, systems engineering, algorithm and software development, and military systems integration. The company, founded in January 2021, also has existing relationships with the Navy and Marines.

“The pairing of academia and industry is highly encouraged by the Department of Defense (DoD). We plan to co-develop new technology with UofSC in the lab and ultimately build a product that the Navy and Marines will use and trust. Beyond the technology, the talent pipeline that’s created is highly attractive to the DoD,” Hartman says.   

Tristan Kyzer, who earned his bachelor’s and master’s degrees from the CEC and was a graduate research assistant for Wang’s group, is now a mechanical engineer at Integer Technologies. But he also served in the U.S. Marine Corps for 10 years and is excited to build a tool that can aid a new generation of Marines.

“I was an infantryman, so much of my time was spent in operational environments similar to the ones this product will be used in,” Kyzer says. “Having the background to understand the operational context for our customer gives me a better understanding of the design space for our product. Firsthand understanding of the conditions will aid in the ability to design a product that the user can rely on.”

Wang hopes this research will be successful and continue securing funding for future research. His goal is to make the university a hub for reliable perception systems research.

“This is only the starting point,” Wang says. “We are confident that this project will produce a knowledge base and technology demonstrations that will make advances critical to providing the state-of-the art capability.”