Credit: CC0 Public Domain

Is it possible some instances of artificial intelligence are not as intelligent as we thought?

Call it artificial artificial intelligence.

A team of computer graduate students reports that a closer examination of several dozen information retrieval algorithms hailed as milestones in artificial research were in fact nowhere near as revolutionary as claimed. In fact, AI used in those algorithms were often merely minor tweaks of previously established routines.

According to graduate student researcher Davis Blalock at the Massachusetts Institute of Technology, after his team examined 81 approaches to developing commonly believed to be superior to earlier efforts, the team could not confirm that any improvement, in fact, was ever achieved.

"Fifty papers in," Blaclock said, "it became clear that it wasn't obvious what the state of the art even was."

Much credit for advances in artificial intelligence over the past decade lies with improvements in hardware such as graphics processors, computer processing units and cameras that allowed for exponential growth in complex search projects, , photography, language translation and voice recognition as well as breakthroughs in ever-more fantastic visualizations of virtual reality games. Algorithmic improvements have certainly assisted as well.

But the MIT team says at least some improvements in AI algorithms have been illusory.

They found, for instance, that with minor tweaks on long-established AI algorithms, the old procedures worked essentially as well as the highly touted "new-and-improved" ones. In several instances, newer AI models were in fact found to be inferior to older approaches.

Credit: X. LIU/SCIENCE; (DATA) MUSGRAVE ET AL., ARXIV: 2003.08505

An article in Science magazine assessing the study cites a meta-analysis of information retrieval algorithms used in search engines over a decade though 2019 and found "the high mark was actually set in 2009." Another study of neural network recommendation systems used by streaming services determined that six of the seven procedures used failed to improve upon the simpler algorithms devised years earlier.

Blalock points to inconsistencies in techniques used to compare algorithms, leaving the accuracy of claims that one approach is better than another open to question.

In fact, it is the inability to properly compare and assess competing approaches that is largely to blame for apparent lack of significant progress in some areas of AI over the past decade, according to one MIT computer scientist. John Guttag, Blalock's Ph.D. adviser, said, "It's the old saw, right? If you can't measure something, it's hard to make it better."

Zico Kolter, a computer scientist at Carnegie Mellon University, speculates that there is greater motivation and social reward to affix one's name to a new than to merely patch and tweak older, established methods.

He studied image-recognition models that were programmed to resist what are called adversarial attacks by hackers. Such an attack uses subtly altered code to bypass system security. An early approach called projected gradient descent (PGD) fended off such attacks by training an AI system to distinguish between authentic and fake examples of code. It was considered a sound approach, but was supposedly bypassed by newer and better protocols. However, Kolter's team of researchers found that a simple tweak on the older PGD approach made it virtually indistinguishable in effectiveness compared with the newer methods.

"It's pretty clear that PGD is actually just the right algorithm," Kolter said. "It's the obvious thing, and people want to find overly complex solutions."

More information: Matthew Hutson. Core progress in AI has stalled in some fields, Science (2020). DOI: 10.1126/science.368.6494.927

Journal information: Science