Research team proposes unclonable, invisible machine vision markers using cholesteric spherical reflectors

Over the last three decades, the digital world that we access through smartphones and computers has grown so rich and detailed that much of our physical world has a corresponding life in this digital reality. Today, the physical and digital realities are on a steady course to merging, as robots, Augmented Reality (AR) and wearable digital devices enter our physical world, and physical items get their digital twin computer representations in the digital world.

These digital twins can be uniquely identified and protected from manipulation thanks to crypto technologies like blockchains. The trust that these technologies provide is extremely powerful, helping to fight counterfeiting, increase supply chain transparency, and enable the circular economy. However, a weak point is that there is no versatile and generally applicable identifier of physical items that is as trustworthy as a blockchain. This breaks the connection between the physical and digital twins and therefore limits the potential of technical solutions.

In a new paper published in Light: Science & Applications, an interdisciplinary team of scientists led by Professors Jan Lagerwall (physics) and Holger Voos (robotics) from the University of Luxembourg, Luxembourg, and Prof. Mathew Schwartz (architecture, construction of the built environment) from the New Jersey Institute of Technology, U.S., propose an innovative solution to this problem where physical items are given unique and unclonable fingerprints realized using cholesteric spherical reflectors, or CSRs for short.

Lagerwall explains, "The unique feature of CSRs is that they are selective retroreflectors, sending light back to a source in any direction, but only within a narrow wavelength band and only with one specific circular polarization. The narrow wavelength band allows us to make the CSRs undetectable to humans by localizing the reflections in the near-infrared or near-ultraviolet regions, easily readable by robots and AR devices but invisible to the human eye."

"The circular polarization allows us to separate out only the CSR signals from any complex background, making any message written using CSRs stand out with extraordinary contrast."

It is by coating surfaces with CSRs laid out in a particular pattern that the team gives physical objects their "fingerprints." Since robots and AR devices can read the fingerprints with exceptional clarity and reliability, their operation becomes much more reliable if the CSR fingerprints encode information about the objects, giving the devices in question a much better understanding of their surroundings.

The ideal pattern in which the CSRs are laid out in the fingerprints is one of so-called fiducial markers, as explained by Voos: "Fiducial markers are square-shaped binary patterns, quite similar to QR codes in their appearance. The particular arrangement of black and white pixels encodes an identity; for instance, telling a machine reading the pattern that this is a door, a car or a wall. And by measuring how large the pattern is and how it is distorted due to the viewing angle from the original square shape, the machine can—with very low computational effort—establish how far away the item carrying the pattern is and how it is oriented with respect to the machine."

Conventional black and white fiducial markers are common in robotics and AR research, but their large size and high-contrast physical appearance make it impossible to deploy them in most human-populated spaces. They must also be illuminated with visible light to be useful. For these reasons, they are today relegated to research labs or restricted areas.

By making the markers using CSRs optimized for near-IR or near-UV operation, their power in assisting robots and AR devices becomes accessible everywhere, including spaces where people work, play or rest, because such CSR markers are invisible to humans and therefore do not disturb the environment as we experience it.

At the same time, the circular polarization of CSR reflections highlights the CSR markers against any background for the machines that need to detect them, removing problems of false positives and difficulties in detection.

The omnidirectional retroreflectivity is key to their usefulness, and Schwartz points out that "retroreflectors are common in motion capture work, whether for tracking humans or robots, because no matter how the subject under study moves, the retroreflector sends the light back to the camera that also sends out the light signal. But regular retroreflectors send back all light, so they are very visible; this is why they are also used in road signs and high-visibility clothing."

"By making fiducial markers using CSRs, a robot or AR device can track objects in its surroundings that carry the markers from any angle, during day or during night, by illuminating the scene with near-IR or near-UV light that is harmless and invisible to humans," Schwartz continues.

A final key advantage of markers made using CSRs is that the exact local arrangement of the individual CSRs is unpredictable, and so difficult to reproduce that every CSR marker is effectively unclonable. Moreover, when investigating a CSR marker at close distance, the appearance depends on how one illuminates and observes it, meaning that a certain marker does not give one pattern upon close inspection, but actually an infinite series of dynamic patterns, unique for each marker yet well defined.

This turns CSR markers into what are referred to as physical unclonable functions, or PUFs, which are very powerful in authenticating physical objects.

This is the final key advantage of CSR markers, says Lagerwall, summarizing, "CSR markers work at multiple scales: From a distance, they reproducibly and reliably give a pattern that we have chosen, specifically the fiducial marker pattern that identifies which category of object is carrying the marker pattern. But when a robot or AR device manipulates an object, it can investigate the marker close up, revealing the locations of individual CSRs and checking how they reflect light that is sent in different directions or over different areas, for instance by turning on and off individual LEDs in a standard illumination ring."

"In contrast to the far-field fiducial marker pattern, the resulting near-field patterns are unique to this particular object, thus also giving its identity. This is how CSR markers allow robots and AR devices to securely link the physical world to the digital twin, because they tell the machine not only what kind of object they are dealing with, but even which specific item this is. And since the optics are outside the visible spectrum, we humans won't notice that anything has changed."