Figure-ground representation in deep neural networks

Brian Hu, Salman Khan, Ernst Niebur, Bryan Tripp

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.

Original languageEnglish (US)
Title of host publication2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728111513
StatePublished - Apr 16 2019
Event53rd Annual Conference on Information Sciences and Systems, CISS 2019 - Baltimore, United States
Duration: Mar 20 2019Mar 22 2019

Publication series

Name2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019


Conference53rd Annual Conference on Information Sciences and Systems, CISS 2019
Country/TerritoryUnited States

ASJC Scopus subject areas

  • Information Systems


Dive into the research topics of 'Figure-ground representation in deep neural networks'. Together they form a unique fingerprint.

Cite this