Human infants’ ability to learn language is an amazing skill that is unrivaled by existing AI systems.
In order for AI to learn and use language in the same way as humans do, it requires massive amounts of data. On the other hand, human infants learn to speak naturally as they grow up. Understanding the mechanisms behind infants’ amazing language learning ability and applying these to artificial intelligence models would enable us to take current AI approaches to the next level. Such neuro- and cognitive science inspired approaches to artificial intelligence are once again attracting interest around the world, and among those proposals to design computational models based on how infants’ language acquisition process are spotlighted as cutting-edge initiatives in this area. However, the complex mechanisms underlying infants’ amazing learning abilities have yet to be understood adequately, as they are to date very difficult to imitate artificially.

So how do babies actually learn a language with such incredible speed and efficiency? It is a known fact that infants’ social environment, for instance the mother, plays a crucial role in the amazing learning skills of babies. However, we have yet to understand the complex mechanism of how such social environmental factors actually influence learning. Our project team aims to establish a theory of early language development by understanding the mechanism of language learning in babies, in particular the role of social interaction in learning.

In this project, we established a unique new experimental method based on two-way gaze-contingent eye-tracking in order to gain an in-depth understanding of the role of social interaction in infant learning. Our experimental system simultaneously tracks the gaze of both the infant participant and an adult interaction partner on their respective screens, thereby recording their gaze interaction. This configuration enables the experimental assessment of the effects of dyadic interaction on learning while maintaining their complex dynamics, but reduced to a two-dimensional manageable format. Moreover, we will be able to evaluate various types of learning sequences with this set-up. For example, we can compare the effect of interactive and non-interactive scenarios, and the effects of such scenarios on learning of cross-modal associations between sounds and objects.

The experimental data generated by this project is expected to not only provide clues for elucidating the dynamics of social interaction in infant language learning but serve as the foundation of agent-based AI systems for modeling and understanding the dynamic characteristics of interactive sequences. Moving forward, it may help establish innovative educational methods for young children and to apply them to interactive digital solutions such as video chat systems.
More speculatively, the unique contribution of this project would be to put the shortcomings of current AI systems into sharper relief: current AI methods rely on decoupled forms of learning, whereas human capacities depend on interactive forms of learning. Accordingly, the project could provide insights that pave the way toward the design of a not only developmentally-inspired, but more interaction-centered form of AI.

In the first part of our project, we established that infants learn better from interactive than non-interactive agents, and that interactivity is an especially important cue for young learners. Using gaze-contingent eye-tracking, we designed a virtual agent that reacted (or did not react) to infants’ gaze, and who taught them new words. Our findings show that young learners are able to learn these new words from the reactive agent, but no the non-reactive agent. We also established that this effect is especially pronounced around the age of 1 year and becomes less relevant with increasing age.
After establishing the importance of interactivity for language learning, the second part of our project sought to assess the mechanisms behind this interactive learning advantage. To this end, we developed a two-way gaze-contingent eye-tracking paradigm, in which infant and experimenter mutually track each other’s eye movements in an abstract, screen-mediated setting. This setup allows to measure the complex dynamics of interaction in real time, but in a tightly controlled experimental settings. We have established that infants can distinguish between interactive and non-interactive events based on the on-screen dynamics alone, and without any other cue to agency. Interactive events lead to more frequent and longer encounters, which is a potential mechanism underlying the social learning advantage.

Future study will link these findings to language learning outcomes, and provide important insights on the design of interactive AI agents for maximizing learning success.