The mystery of visual perception

TU Professor Thomas Wallis receives ERC Consolidator Grant


How do we perceive the individual objects in a scene even if they are only partially visible? And how do we learn this so-called segmentation from visual experience? This is now being investigated by researchers at TU Darmstadt. The European Research Council is funding the project with a renowned ERC Consolidator Grant to the tune of approximately two million euros.

Professor Wallis' lab uses high-precision display and eye-tracking systems to conduct experiments on human behaviour.

A photo shows us only the bonnet of a car protruding from behind a house – and yet adult viewers instantly see that it is a car. This is because the human mind can normally grasp what objects are present in a scene, and also imagine the hidden parts of it. But how exactly does the mind recognise the meaning of scenes like this? This question will be investigated in the “SEGMENT” research project headed by TU Psychology and Cognitive Science professor Thomas Wallis, which is now funded by a renowned ERC Consolidator Grant. The funding amount over a period of five years amounts to around 2.1 million euros.

The first step in the visual understanding of scenes is known as segmentation. This is when the human brain attempts to recognise which parts of a scene belong to which objects. Adults can do this to a certain extent from photos – but this is likely because they learnt to see in a 3D world as toddlers rather than purely through photos. Information on our environment is provided by, for instance, the way scenes project into the human eye, how light is influenced by the optics of the eye, how photoreceptors in the eye react to light, and how the eyes move.

How do adults segment scenes

However, it is not known how people combine all this information from active 3D vision for segmentation. One reason is because traditional visual display devices are unable to accurately mimic these factors – and it would be unethical to manipulate these factors for a long time in young children. This is where Professor Wallis's project comes in. The aim of “SEGMENT” is to develop a new display device and experimental methods to investigate how adults segment scenes, and to create simulations of the visual system from early development to adulthood. Professor Wallis's team at Darmstadt wants to use groundbreaking new technologies based on advanced computer graphics and machine learning for this purpose.

There are also plans to conduct computer experiments with artificial neural networks to help in understanding segmentation learning. The researchers want to systematically restrict or manipulate various factors such as optics or eye movements. At the end, the learned behaviour of the artificial networks will be compared with the behaviour of adults during segmentation in the active exploration of 3D scenes. “We want to use similarities and differences to better understand a fundamental mystery in perception: how the mind makes sense of scenes,” explained Professor Wallis.

About the recipient

Professor Thomas Wallis, Ph.D.
Professor Thomas Wallis, Ph.D.

Thomas Wallis has been a Professor of Perception at the Institute of Psychology and Centre for Cognitive Science at TU Darmstadt since 2021. His research focuses on visual perception in humans and machines, machine learning and cognitive modelling, and applications in visual perception.

Australian-born Wallis received his doctorate from the University of Queensland in Australia in 2010. He then conducted research at Schepens Eye Research Institute and Harvard Medical School in the USA as well as at the Centre for Integrative Neurosciences and the Department of Computer Science at the Eberhard Karl University of Tübingen. Thomas Wallis was a researcher at Amazon Deutschland from 2019 until 2021.


ERC Consolidator Grants are awarded by the European Research Council to researchers from all disciplines between seven and twelve years after completing their doctorates. The European Union does so to promote promising research: the Consolidator Grant is aimed at researchers who have already achieved excellent work and are now to be supported in groundbreaking research projects in order to achieve scientific consolidation. In the current round, 321 grants have been awarded and 2.222 applications submitted.

Along with Professor Wallis, other recipients of an ERC Consolidator Grant at TU Darmstadt were Professor Leopoldo Molina-Luna for the ELECTRON project and Professor Eva Kassens-Noor for the scAInce project .