United States

We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on the spoken language comprehension. We compare the ERP signature (N400) associated to each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (i.e., measures of how predictable words in their linguistic context) generated on the basis of different types of language models (n-gram and transformer models) predict N400 responses for each word. We find that cognitive effort (measured by surprisal) in a multimodal setting significantly differs from that in a unimodal setting. The analysis of surprisal estimates from language models reveals that while Transformer-based models provide better fit to N400 signals in audio-only setting, 2-gram language models fit better in the multimodal setting, suggesting that local verbal context significantly influences cognitive processing effort in a multimodal setting.


CogSci 2023

Are words equally surprising in audio and audio-visual comprehension?

technical paper

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

To investigate how co-speech gestures modulate linguistic understanding, we experimentally test two Bayesian Pragmatic models. We identify the semantic effect of a spoken or gestural utterance with the change it makes in listener’s probabilistic predictions of the speaker’s communicative intentions. We focus on action-expressing gestures and the respective verbs that correspond to action-affording instruments or, respectively, their denoting nouns. Combining Pustejovsky’s’ Generative Lexicon approach with Gibson’s affordance theory, we ask: (1) Does a co-speech gesture make any difference for semantic comprehension and the corresponding probabilistic prediction? (2) Is the semantic effect of a gesture similar or identical to the one of the corresponding verb? (3) To which extent does the gesture’s semantic effect depend on the listener’s recognition of the gesture as an expression of the corresponding verb? (4) Does the comprehended affordance predict the instrument better than the co-occurrence statistics (GloVe) regarding the verb and the noun?


Modelling the Integration of Co-Speech Gestures into Sentence Meaning Composition

Moral learners face an underdetermination problem – the rules they are taught cannot account for all novel cases. One way learners solve this problem is through closure principles, through which a learner assumes that anything that isn’t explicitly forbidden is permitted, and vice versa. The current work aims to explore whether closure rules are used when reasoning about obligations. Building on previous work, we ask 1) whether adults and children use obligation rules in a similar manner to rules about permissibility and impermissibility, and 2) how early in life these inference abilities emerge. Across two studies, we explore inferences about obligation from both adults (N = 120, Mage = 33.73 years) and children (N = 103, Mage = 5.52 years). We found that while both adults and children rationally learn closure principles consistent with deontic logic, children and adults make opposite inferences about novel cases when provided with obligation rules.


Underdetermination and Obligation Rules: Adult and Children’s use of Closure Principles in Moral Learning

A growing body of literature has revealed ordinary causal judgement to be sensitive to normative factors, such that a norm violating agent is regarded more causal than their non-norm violating counterpart. The present paper aims to tease apart two competing explanations: the Responsibility View and the Bias View. The Bias View, but not the Responsibility View, predicts features peripheral to the agent’s responsibility to impact causal attributions. In a series of three preregistered experiments (N = 1162), we present new evidence that the Norm Effect arises from such peripheral features, namely from nonpertinent or entirely silly norm violations. Further, we show that this effect cannot be explained by recourse to the agent’s foreknowledge or desire of the outcome, nor by its foreseeability: the Norm Effect arises even when participants judge the norm violating agent’s doing as equally foreseeable. This, we argue, provides evidence in favour of the Bias View.


Causation, Foreseeability, and Norms

Human learning is often motivated by self-imposed challenges, which guide behavior even in the absence of external rewards. Previous studies have shown that humans can use personal goals to “hack” the definition of reward, warranting an extension of the classic reinforcement learning framework to account for the flexible attribution of value to outcomes according to current goals. However, previous studies have shown that learning through goal-derived outcomes is less efficient than learning through more established reinforcers, such as numeric points. At least three possible explanations exist for this sort of impairment. First, such differences may be due to occasional lapses in executive function, which is required to encode and recognize goals, resulting in subsequent failure to update values accordingly. Second, the higher working memory load required to encode one-shot, novel stimuli as desirable outcomes may impair people’s ability to update and remember correct stimulus-reward associations. Third, a weaker commitment to arbitrary goals may result in dimmer appetitive signals. By extending existing experimental paradigms that include learning from both familiar rewards and abstract, goal-contingent outcomes and combining them with computational modeling techniques, we find evidence for each of the proposed accounts. While other factors might also play a role in this process, our results provide an initial indication of the key elements supporting (or impairing) the attribution of rewarding properties to otherwise neutral stimuli, which enable humans to better pursue arbitrarily set goals.


Human hacks and bugs in the recruitment of reward systems for goal achievement

Why do babies look longer when they see an object pass through a solid wall, or a person act inefficiently, during violation-of-expectation (VOE) studies? Here we test two non-mutually exclusive hypotheses: (i) VOE involves domain-general processes, like visual prediction error, and curiosity about the source of surprise. (ii) VOE involves domain-specific processes, like prediction error over distinctively physical and psychological expectations (objects fall; agents behave rationally). In a pre-registered experiment, we scanned 32 adults using functional magnetic resonance imaging (fMRI) while they watched videos of agents and objects, adapted from infant behavioral research. Early visual regions responded equally to surprising and expected events in both domains, providing evidence against domain-general visual prediction error. Some multiple demand regions, that are engaged when people deploy goal-directed attention, responded more to surprising events from both domains, providing evidence for domain-general endogenous attention. Domain-specific regions, that prefer stimuli involving agents vs objects more broadly, showed similar preferences for the current videos of agents and objects. One region implicated in physical reasoning responded selectively to unexpected events from the physical domain, providing evidence for domain-specific physical prediction error. Thus, in adult brains, both domain-specific and high-level domain-general regions encode violation-of-expectation involving agents and objects, paving the way towards future developmental work.


Domain-specific and domain-general neural responses to surprising psychological and physical events

This paper presents a method for reproducing a simple central pattern generator (CPG) using a modified Echo State Network (ESN). Conventionally, the dynamical reservoir needs to be damped to stabilize and preserve memory. However, we find that a reservoir that develops oscillatory activity without any external excitation can mimic the behaviour of a simple CPG in biological systems. We define the specific neuron ensemble required for generating oscillations in the reservoir and demonstrate how adjustments to the leaking rate, spectral radius, topology, and population size can increase the probability of reproducing these oscillations. The results of the experiments, conducted on the time series simulation tasks, demonstrate that the ESN is able to generate the desired waveform without any input. This approach offers a promising solution for the development of bio-inspired controllers for robotic systems.

Generating oscillation activity with Echo State Network to mimic the behaviour of a simple central pattern generator

According to the embodied approach to cognition, perception and action are tightly intertwined, as perception is for action and is guided by action. To better understand what this view implies behaviorally, we studied how active movement and intentionality during perceptual exploration affect perceptual accuracy. Participants explored two-dimensional objects using a sensory substitution device, then reported their object size estimates. We manipulated 1) their control over exploratory movements as being either Active (control present) or Passive (control absent) and 2) their knowledge of the task goals, being either Specific (task-focused) or Generic. We found no difference between the Active and Passive conditions but significantly higher perceptual accuracy in Specific Intention trials compared to Generic Intention ones. These results clarify the nature of ''active'' perception and contribute to the growing body of evidence that ''higher level'' cognitive goals shape how we dynamically sample even ''low level'' sensory information from the world.


Uncovering the Role of Intention in Active and Passive Perception

Human cognition is shaped by our bodies and actions. The influence of embodiment on cognition is particularly crucial during early development. Recent evidence shows that infants use actions to accomplish cognitive and social tasks that may later be solved internally. In our study, we propose that a sensorimotor mechanism to hand-eye coordination is through a full path from manual action, to visual saliency, and to visual attention. To provide a rigorous test of this pathway, we analyzed multimodal behavioral data collected from parent-infant toy play. We focused on linking infants’ manual actions with visual properties in the infant’s view and attention. Further, we extended our analyses to quantify the effects of manual actions on one’s own visual attention, infant’s actions on parent attention, and parent’s actions on infant attention. Results suggest that both infants’ and parents’ actions create visual saliency of objects to support visual attention and joint attention.

Using manual actions to create visual saliency: an outside-in solution to sustained attention and joint attention

This paper presents interdisciplinary and participatory research conducted with laryngectomees with an aim to widen research paradigms for cognitive sciences. Studying daily communication in an interactive context of people living without larynx, we put special emphasis on methodology combining engaged epistemology with ethnographic fieldwork, novel in the domain of this specific perturbation. Results valuable for designers, therapists and research participants themselves demonstrate that the greater awareness is needed of i) influences of anatomical and physiological adaptations on communication and social relations, ii) multimodal and context-dependent alternative strategies of conversation, iii) the crucial role of participants’ agency. We claim that some of our unexpected observations were possible, because of choosing the research approach that extended our epistemological capacities, namely by remaining open to novel interpretive categories and perspectives. We describe how dialogue between epistemologically engaged cognitive science and ethnographic fieldwork facilitated our understanding of the problem in a specific case.

Premium content

Are words equally surprising in audio and audio-visual comprehension?

Downloads

Next from CogSci 2023

Modelling the Integration of Co-Speech Gestures into Sentence Meaning Composition