OPTIMISM IS HARD to come by among those trying to build artificial intelligence safely. An abiding fear is that neural networks—the complex structures that underpin most modern AI systems—are largely inscrutable. Understanding how part of a neural network functions, in other words, does not always allow engineers to predict a model’s behaviour. If models continue to become ever more capable without being easier to interpret, say researchers, the chances of apocalyptic malfunctions will grow too.
But on October 15th Nick
OPTIMISM IS HARD to come by among those trying to build artificial intelligence safely. An abiding fear is that neural networks—the complex structures that underpin most modern AI systems—are largely inscrutable. Understanding how part of a neural network functions, in other words, does not always allow engineers to predict a model’s behaviour. If models continue to become ever more capable without being easier to interpret, say researchers, the chances of apocalyptic malfunctions will grow too.
But on October 15th Nick Cammarata, a researcher at Goodfire, an American AI startup, published an essay offering some hope. He argued that neural networks’ apparent opacity is largely down to bad tools and a failure of imagination.
To illustrate this, he used an often-cited example of a model designed to recognise images. One of its neurons fired in response to foxes’ faces as well as the fronts of cars, suggesting the model packs concepts together arbitrarily in a manner humans have no hope of understanding. Mr Cammarata reinterpreted the neuron’s role as identifying pointy objects: it also fires for cats’ faces and pool tables, but not for a dobermann’s mug or a snail’s shell.
Mr Cammarata applied his new insights to the innards of GPT-2 XL, an early neural network made up of 307,200 neurons. In 2023, while working at OpenAI, the model’s maker, he had managed to interpret only 1.7% of its neurons. Working alongside modern AI tools, Mr Cammarata now reckons about 70% of its neurons in a random sample of 500 have just one main meaning. That may be good news for interpreting what goes on under the hood of AI models. “I think this may be remembered as a major moment in scientific history,” wrote an anonymous OpenAI employee.
Not everyone is convinced. Neel Nanda, an AI researcher at Google DeepMind, worries that, although it may be possible to make sense of some of what a neuron does, it will be hard to guarantee a complete picture. A neuron that recognises pointed objects, for example, could be part of a plan to take over the world when fired in combination with one representing socks. It is also too early to tell whether Mr Cammarata’s approach will scale to the millions of neurons that comprise frontier models. For now it offers a mere glimmer of light amid the AI gloom.
One Subscription.
Get 360° coverage—from daily headlines
to 100 year archives.
Archives
HT App & Website