United States

We present a novel method for image saliency prediction that leverages a cognitive model of visual attention as an inductive bias.
This is in stark contrast to recent purely data-driven
models that have achieved performance improvements mainly by increased model capacity, resulting in high computational costs and the need for large scale, domain specific training data. 
We demonstrate that by leveraging a cognitive model of visual attention, our method achieves competitive performance to the state-of-the-art across several natural image datasets while only requiring a fraction of the parameters.
Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization.
We further provide large-scale cognitively plausible synthetic gaze data on corresponding images in the full MSCOCO and FigureQA datasets, which we used for pre-training.
These results are highly promising and underline the significant potential of bridging between first-principle cognitive and data-driven models for computer vision tasks, potentially also beyond saliency prediction, and even visual attention.

\bf Keywords: neural networks, computational models of cognition, computer vision, saliency prediction, human visual attention

CogSci 2023

Improving neural saliency prediction with a cognitive model of human visual attention

poster

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Visual emergence is the phenomenon in which the visual system obtains a holistic perception after grouping and reorganizing local signals. The picture Dalmatian dog is known for its use in explaining visual emergence. This type of image, which consists of a set of discrete black speckles, is called an emerging image. Although Gestalt theory summarizes perceptual organization into several principles, it remains ambiguous how these principles affect the perception of emerging images. This study, therefore, designed three psychological experiments to explore the factors that influence the perception of emerging images. In the first, we found that the density of speckles in the local area and the arrangements of some key speckles played a key role in the perception of an emerging case. We set parameters in the algorithm to characterize these two factors. We then automatically generated emerging-test images through the algorithm and verified their effectiveness in two subsequent experiments.


Important Clues that Facilitate Visual Emergence: Three Psychological Experiments

Predictive coding has emerged as a prominent model of how the brain learns through predictions and prediction errors. Traditional predictive coding focused primarily on sensory coding and perception. Here we propose active predictive coding (APC), a unified framework for perception, action and cognition. By learning hierarchical world models, the APC framework addresses important open problems in cognitive science and AI such as: (1) how do we learn compositional representations, e.g., part-whole hierarchies for equivariant vision? and (2) how do we solve large-scale planning problems, which are hard for traditional reinforcement learning, by composing complex action sequences from primitive policies? APC exploits hypernetworks, self-supervised learning and reinforcement learning to learn hierarchical models that combine task-invariant state transition networks and task-dependent policy networks at multiple abstraction levels. We illustrate the applicability of the APC model to visual perception using a variety of vision datasets as well as to a scalable hierarchical planning problem. Our results represent, to our knowledge, the first demonstration of a unified approach to addressing the part-whole learning problem in vision, the nested reference frames learning problem in cognition, and the integrated state-action hierarchy learning problem in reinforcement learning.

Active Predictive Coding: A Unified Neural Framework for Learning Hierarchical World Models for Perception and Planning

Pragmatics is central to natural language, enabling speakers to communicate efficiently by using structures like ellipsis and anaphora to shorten utterances without loss of meaning. These structures require a listener to interpret an ambiguous form - like a pronoun - and infer the speaker’s intended meaning—who that pronoun refers to. Despite this potential to introduce ambiguity, anaphoric structures are ubiquitous across human language. In an effort to better understand how such structures emerge, we look to see if analogous structures can emerge in languages used between artificial neural networks that are trained to solve a communicative task. We show the following: first, despite the potential for increased ambiguity, languages with anaphoric structure are highly learnable by neural models. Second, anaphoric structure emerges naturally without needing to impose additional constraints on the networks. Finally, we introduce an explicit efficiency pressure on the speaker and show that this increases the prevalence of anaphoric structures. We conclude that certain pragmatic structures straightforwardly emerge between neural networks without explicit pressures, but that the balance between speaker and listener needs conditions the degree and nature of their emergence.

Anaphoric Structure Emerges Between Neural Networks

Experimental research on the processing and acquisition of Scalar Implicatures (SIs) relies on behavioral tasks that purport to measure the rate at which SIs are computed within an experimental paradigm. Two paradigms, the Truth Value Judgment Task (TVJT) (Gordon, 1998; Crain & Thornton, 2000) and the Picture Selection Task (PST) (Gerken & Shady, 1998) have dominated the experimental pragmatics literature; however, there has been little investigation of the way in which the selection of the experimental paradigm may affect the estimates of implicature rate. Here we report the results of two studies testing participants in both TVJT and PST using three different linguistic scales in English: “ad-hoc”, “or-and”, and “some-all”. In the first experiment, the task variation was manipulated within subjects while in the second experiment, it was manipulated between subjects. We found that the estimated rate of SI computation varied noticeably between these tasks for two out of the three linguistic scales. This suggests that the experimental paradigm itself has a significant impact on our estimates of the implicature rate and consequently psycholinguistic theories of implicature computation.

Task Variation in Scalar Implicature Computation

Utterances mentioning a highly predictable event are known to elicit atypicality inferences (Kravtchenko and Demberg, 2015; 2022). In those studies, pragmatic inferences are measured based on typicality ratings. It is assumed that comprehenders notice the redundancy and "repair'' the utterance informativity by inferring that the mentioned event is atypical for the referent, resulting in a lower typicality rating. However, the actual inferences that people make have never been elicited. We extend the original experimental design by asking participants to explain their ratings and administering several individual differences tests. This allows us to test (1) whether low ratings indeed correspond to the assumed inferences (they mostly do, but occasionally participants seem to make the inference but then reject it and give high ratings), and (2) whether the tendency to make atypicality inferences is modulated by cognitive factors. We find that people with higher reasoning abilities are more likely to draw inferences.

What inferences do people actually make upon encountering informationally redundant utterances? An individual differences study

When and how do people detect and adjust for sample bias? The present work examines how people use a small, potentially biased sample to reason about the population from which the sample was drawn. We developed a task to explore when and how people generalize from selectively sampled evidence and which cues learners use to infer the presence of selection bias and adjust their responses accordingly. Participants sampled data that followed several different distributions (e.g., bimodal), and constructed their frequency distribution of a larger “population”. Participants “filled in” missing data when sample distribution observations were sparse or were inconsistent with priors about distribution shape. Human responses were compared with several computational model predictions. An averaging model captured response curves better than a normative Bayesian approach (for some distributions). Results suggest that people draw on prior distributional expectations and observed sample distributions when generalizing from potentially biased samples.

Censor Detection: Detecting and adjusting for sample bias

It is generally difficult for people to make rational and accurate judgments under their limited cognitive resources. In this study, we propose an intervention to easily improve people’s judgment accuracy with less workload by waiting for a short time at the beginning of a task. By using a simple binary choice task, we found that when a short (1s) waiting time was inserted, participants showed higher accuracy than when no waiting time was inserted, and they felt less mental workload than when a longer (2.5s) waiting time was inserted. To examine the underlying implicit cognitive processes, we applied mouse tracking approaches during choice behaviors. We found that the inserted time enhanced participants’ change of mind (i.e., they amended their initial wrong judgments). These results suggest that making people wait for only 1s will serve as a simple, effective, and resource-rational intervention to boost people’s accuracy of judgments. Because of its simplicity, we believe that this intervention can be applied in various fields.

A one-second wait improves judgment accuracy: A mouse tracking reveals cognitive processes during choice behaviors

Virtual Reality (VR) presents a novel method to study the mind by generating dynamic, immersive environments for participants to engage with. However, the adoption of VR in cognitive science has been slow---for one, in order to use VR as a methodological tool, we need to verify that studies conducted in VR reflect the same assumptions that are common in existing experimental methods. Here, we investigate this question by drawing on existing empirical work in the domain of intuitive physics. In Experiment 1, we found that participants' judgments on a physical scene prediction task largely correlated with the same judgments collected using conventional methods. In Experiment 2, participants preferentially explored objects that violated core principles of physics, even engaging in distinct hypothesis-testing behaviors in accordance with the violation. In sum, these findings suggest that VR can be an exciting, novel method to study human behavior.

Intuitions about physical scenes and objects are preserved in Virtual Reality (VR)

Words express various sensory information to various degrees. Norming studies have collected native speakers’ subjective perceptual strength ratings for numerous words in several languages. This paper presents perceptual strength norms for 510 Japanese words, including iconic lexemes called ideophones. The newly collected norms replicated some previously reported correlations, such as the one between certain sensory modalities and iconicity. A systematic comparison between the Japanese and English perceptual strength norms further revealed that Japanese onomatopoeic ideophones tend to be more multisensory than their English equivalents and that Japanese words in general tend to encode more interoceptive information than English words. These findings suggest the usefulness of norming data in linguistic typology.

Perceptual Strength Norms for 510 Japanese Words, Including Ideophones: A Comparative Study with English

How similar are people's meanings of common words? Do differences in word meanings lead to miscommunication? We examined divergence between word meanings using similarity ratings such as "Is a penguin more similar to a chicken or a dolphin". We found that given the identical instructions some people prioritized taxonomic relationships more than other people. Moreover, this taxonomic bias generalized from animals to artifacts suggesting a more general difference in semantic organization. When people with different biases are paired in a matcher-director type task, they were less likely to achieve communicative success.

Premium content

Improving neural saliency prediction with a cognitive model of human visual attention

Next from CogSci 2023

Important Clues that Facilitate Visual Emergence: Three Psychological Experiments