The mystery of visual perception

A photo shows us only the bonnet of a car protruding from behind a house – and yet adult viewers instantly see that it is a car. This is because the human mind can normally grasp what objects are present in a scene, and also imagine the hidden parts of it. But how exactly does the mind recognise the meaning of scenes like this? This question will be investigated in the “SEGMENT” research project headed by TU Psychology and Cognitive Science professor Thomas Wallis, which is now funded by a renowned ERC Consolidator Grant. The funding amount over a period of five years amounts to around 2.1 million euros.

The first step in the visual understanding of scenes is known as segmentation. This is when the human brain attempts to recognise which parts of a scene belong to which objects. Adults can do this to a certain extent from photos – but this is likely because they learnt to see in a 3D world as toddlers rather than purely through photos. Information on our environment is provided by, for instance, the way scenes project into the human eye, how light is influenced by the optics of the eye, how photoreceptors in the eye react to light, and how the eyes move.

However, it is not known how people combine all this information from active 3D vision for segmentation. One reason is because traditional visual display devices are unable to accurately mimic these factors – and it would be unethical to manipulate these factors for a long time in young children. This is where Professor Wallis's project comes in. The aim of “SEGMENT” is to develop a new display device and experimental methods to investigate how adults segment scenes, and to create simulations of the visual system from early development to adulthood. Professor Wallis's team at Darmstadt wants to use groundbreaking new technologies based on advanced computer graphics and machine learning for this purpose.

There are also plans to conduct computer experiments with artificial neural networks to help in understanding segmentation learning. The researchers want to systematically restrict or manipulate various factors such as optics or eye movements. At the end, the learned behaviour of the artificial networks will be compared with the behaviour of adults during segmentation in the active exploration of 3D scenes. “We want to use similarities and differences to better understand a fundamental mystery in perception: how the mind makes sense of scenes,” explained Professor Wallis.

mih