For all the impressive tasks that drones can do, there is one that remains beyond their power: facial recognition. Drones are generally much farther from their subjects than the kind of cameras, such as CCTVs, that are ordinarily used for biometrics. At these distances a face may consist of only a few dozen pixels. Atmospheric turbulence caused, for example, by rising hot air, can distort features like the distance between one’s eyes. And because they record from the sky, drones’
For all the impressive tasks that drones can do, there is one that remains beyond their power: facial recognition. Drones are generally much farther from their subjects than the kind of cameras, such as CCTVs, that are ordinarily used for biometrics. At these distances a face may consist of only a few dozen pixels. Atmospheric turbulence caused, for example, by rising hot air, can distort features like the distance between one’s eyes. And because they record from the sky, drones’ on-board cameras may capture only a partial view of a face (or, if someone is wearing a wide-brimmed hat, none at all).
But new technology from a team at Michigan State University (MSU) seeks to change all that and extend the spying powers of artificial intelligence (AI) into the skies. The system, known as FarSight, suggests that long-range aerial surveillance could soon become far more accurate—and intrusive—than ever before.
The project is funded by the Intelligence Advanced Research Projects Activity (IARPA), part of the American government responsible for marshalling fanciful spy-gadget ideas into real-world use. Other current IARPA projects include an effort to build a device that can modulate a voice in real-time in order to avoid detection by speech-recognition tools, as well as an initiative to make snooping devices small and pliable enough to be woven directly into clothing.
FarSight works with a similarly crafty technique known as “whole-body biometric recognition”. Rather than trying to recognise a subject from their face alone, the system uses a combination of biometric-recognition algorithms that run in parallel.
One set of algorithms discerns a person’s gait. Another generates a 3D reconstruction of their body. Xiaoming Liu, a professor of computer science and engineering at MSU who leads the project, likens it to essentially undressing subjects in order to generate an accurate model of their anatomy, regardless of what they happen to be wearing.
A third set of algorithms runs the subject’s face through a turbulence model that seeks to undo the refractive effects of the choppy air on the light that comes into the camera. This returns the image gathered by the drone to a simulated undistorted state, from which a detailed mapping of the subject’s features is then extracted.
Once captured, the three biometric markers—gait, body shape and face—are fused into a combined profile. This profile can be matched to those of known individuals or, if the target is new, saved for future matching. The entire operation happens in about a third of a second, says Dr Liu. His team is working to scale down the system so that it could fit on a quadcopter
Though it is still an experimental system, FarSight’s early results are impressive. The National Institute of Standards and Technology (NIST), America’s standards body, which has been rating facial-recognition systems for more than a decade, tested FarSight on a set of tricky low-resolution images and videos collected at hundreds of metres, in many cases from a high angle. FarSight outperformed all other systems tested on the same set.
The project also illustrates how large vision models (LVMs), a variant of large language models, could be useful for surveillance. The MSU team used CLIP, a model made by OpenAI, to annotate images of thousands of subjects with textual descriptions—“Muscular-slender, long torso”, “Short torso”, “High-waisted”, “Low-waisted”—which it then used to train the body-shape reconstruction system. The gait-recognition feature is based on a different large vision model, called DINOv2, which was released last year by Meta.
Dr Liu says that LVMs could have even broader applications in the years ahead. This is because LVMs achieve high-performance recognition without requiring big training-data sets, which are difficult and expensive to produce. FarSight’s training and testing data, much of which were collected by Oak Ridge National Laboratory, a scientific-research facility, consists of 876,000 videos and photos of about 3,000 subjects. An LVM, by comparison, is already pre-trained on billions of images and videos, and may require only minimal fine-tuning before it can be used for surveillance-video analytics.
FarSight is not yet ready for the field. Although it outperforms other systems on long-range recognition, its accuracy, as well as its false-positive and false-negative rates, remain “far behind the performance that would be required to make the system deployable”, says Josef Kittler, a professor working on biometrics and computer vision at the University of Surrey, who was not involved in the research.
The system has other limitations, too. A person’s gait, for instance, can change drastically if they carry a heavy load or have suffered an injury, says Dr Kittler. In a second NIST test on a wider data set, FarSight was not the best performer. Dr Liu says that FarSight’s performance also drops considerably if the camera’s angle is very high, if the weather is warm (which causes fiercer turbulence) or if the distance to the target exceeds a kilometre.
Should these shortcomings be resolved, though, it is not hard to imagine how such a system might end up overhead. According to contracting documents, IARPA is looking to create a technology not only for drones but any high or distant camera, such as those mounted on tall buildings or border-surveillance towers. The agency has noted that the outcomes of the programme—which is known as BRIAR and has at least one other active research team, led by an American company called Science and Technology Research—are also intended for “protection of critical infrastructure and transportation facilities”.
“That implies its routine use on civilian populations,” says Jay Stanley at the American Civil Liberties Union, and “creates serious risks of abuse and chilling effects on people’s sense of freedom”. In cities like London or New York, where CCTV cameras are ubiquitous but not contiguous, whole-body recognition could help track individuals across long distances. Veritone, an American company, already markets a system that can match individuals according to such attributes as body shape and hairstyle. In May the company’s boss, Ryan Steelberg, told MIT Technology Review, a magazine, that the tool could be useful in cities where facial recognition is banned.
Those who take matters of privacy into their own hands might also find their personal powers of evasion diminished in the face of whole-body biometrics. Someone wishing to evade a tool like FarSight could no longer just rely on an outfit change or a big hat. They would also need to adjust their gait and, somehow, present a different body shape to cameras. In other words, Dr Liu says, “the whole enchilada”.
Curious about the world? To enjoy our mind-expanding science coverage, sign up to Simply Science, our weekly subscriber-only newsletter.
One Subscription.
Get 360° coverage—from daily headlines
to 100 year archives.
Archives
HT App & Website