Deep Linear Neural Networks and Language Evolution Enables Human-Like Generalization

What is it about?

By modelling how language evolves across generations of learners, we investigate how structured communication systems can emerge and support human-like reasoning. The work is inspired by theories of child development suggesting that language becomes more organized over time as it is passed between learners. To study this process, we analyse linear neural networks trained through “iterated learning,” where each generation of networks learns from the outputs produced by earlier generations. Over time, this process refines language into more compositional forms, where complex meanings can be constructed by combining simpler parts. Using mathematical analyses of shallow and deep neural networks, we show that deep architectures are particularly important for exploiting this compositional structure and achieving systematic generalization—the ability to apply previously learned knowledge to entirely new situations. This is a defining feature of human cognition and language, allowing people to understand and produce sentences they have never encountered before. Our results confirm that multiple generations of learning are required before this structured behaviour fully emerges and that the resulting systems can outperform standard training approaches. At the same time, the work reveals important limitations of iterated learning. While networks can discover regularities in language outputs, they struggle to systematically ignore irrelevant features in the inputs unless they are trained at very large scales. We therefore introduce the idea of “weak systematic generalization” to describe how more robust and human-like forms of generalization gradually emerge as vocabularies and datasets grow. Overall, this research helps connect theories of human language development with modern artificial intelligence, providing insight into both the origins of language structure and the design of more flexible, adaptive AI systems.

Featured Image

Photo by Joshua Fernandez on Unsplash

Photo by Joshua Fernandez on Unsplash

Why is it important?

A central challenge in both cognitive science and artificial intelligence is understanding how humans can learn language so quickly and flexibly from limited experience. This work provides a theoretical explanation for how structured, compositional language can emerge through repeated learning across generations, and why deep neural architectures are critical for exploiting this structure. The results help bridge neuroscience, linguistics, and machine learning by showing how systematic generalization—a hallmark of human cognition—can arise in artificial systems. Importantly, the work also identifies fundamental limitations of current approaches, demonstrating that some forms of generalization only emerge at very large scales. These findings contribute to our understanding of human language development while also informing the design of more robust and adaptable AI systems.

Perspectives