A new vehicle search system for video surveillance networks

A team of researchers at JD AI Research and Beijing University have recently developed a progressive vehicle search system for video surveillance networks, called PVSS. Their system, presented in a paper pre-published on arXiv, can effectively search for a specific vehicle that appeared in surveillance footage.

Vehicle search systems could have many useful applications, including enabling smarter transportation and automated surveillance. Such systems could, for instance, allow users to input a query vehicle, search area and time interval to find out where the vehicle was located at different times during the day.

Existing vehicle search methods typically assume that all vehicle images are cropped well from surveillance videos, using visual attributes or license plate numbers to identify the target vehicle within these images. These approaches mainly focus on content-based vehicle matching, also known as vehicle re-identification (Re-Id).

In recent years, hand-crafted descriptors and convolutional neural networks (CNNs) have greatly enhanced these methods. Nonetheless, identifying a specific vehicle based solely on attributes can be a very challenging task due to intra-instance changes in different cameras and inter-instance differences between similar vehicles. In some cases, license plates can also be misrecognized in surveillance images due to low resolution and noise.

"A complete vehicle search system should consider the problems of vehicle detection, representation, indexing, storage, matching, and so on," the researchers explain in their paper. "Besides, attribute-based search cannot accurately find the same vehicle due to intra-instance changes in different cameras and the extremely uncertain environment."

PVSS, the progressive vehicle search system developed by the researchers, addresses the limitations of current approaches. The system is composed of three key modules: the crawler of vehicle data, the vehicle indexer based on multi-grained features and the progressive vehicle searcher.

"To guarantee high accuracy and efficiency during search, a series of data structures are designed for the vehicle search system," the researchers write in their paper. "In the crawler, not only visual contents but also contextual information are extracted from the surveillance networks. The multimodal data is then exploited by deep learning-based models to obtain discriminative and robust features of vehicles, which are then organized by the multi-level indexes. In the search process, the vehicle is searched in a progressive manner, including the from-coarse-to-fine search in the feature domain and the from-near-to-distant search in the physical space."

Essentially, the vehicle crawler component detects and tracks vehicles in surveillance videos, transferring captured vehicle images, metadata and other contextual information to the cloud or server. Subsequently, the vehicle indexer component extracts and indexes multi-grained attributes of the vehicles, such as visual features and license plate fingerprints.

A query that includes an input vehicle image, as well as the time range and spatial scope of interest is then fed to the vehicle searcher component, which progressively searches for the vehicle in the image database.

The researchers evaluated their progressive vehicle search system on the VeRi dataset, which contains over 50,000 images collected from 20 surveillance cameras in real-world settings. In these tests, PVSS attained remarkable results, outperforming all the appearance-only search methods and multi-modal methods that it was compared to.

"Extensive experiments on a large-scale vehicle search dataset collected from real-world surveillance network shows the state-of-the-art results of the proposed system," the researchers write in their paper.