United States

Understanding the extent to which the perceptual world can be recovered from language is a fundamental problem in cognitive science. We reformulate this problem as that of distilling psychophysical information from text and show how this can be done by combining large language models (LLMs) with a classic psychophysical method based on similarity judgments. Specifically, we use the prompt auto-completion functionality of GPT3, a state-of-the-art LLM, to elicit similarity scores between stimuli and then apply multidimensional scaling to uncover their underlying psychological space. We test our approach on six perceptual domains and show that the elicited judgments strongly correlate with human data and successfully recover well-known psychophysical structures such as the color wheel and pitch spiral. We also explore meaningful divergences between LLM and human representations. Our work showcases how combining state-of-the-art machine models with well-known cognitive paradigms can shed new light on fundamental questions in perception and language research.

CogSci 2023

What Language Reveals about Perception: Distilling Psychophysical Knowledge from Large Language Models

technical paper

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Spurious correlations were found to be an important factor explaining model פerformance in various NLP tasks (e.g., gender or racial artifacts), often considered to be ''shortcuts'' to the actual task.
However, humans tend to similarly make quick (and sometimes wrong) predictions based on societal and cognitive presuppositions.
In this work we address the question: 
can we quantify the extent to which model biases reflect human behavior?
Answering this question will help shed light on model performance and provide meaningful comparisons against humans. 
We approach this question through the lens of the dual-process theory for human decision-making. This theory differentiates between an automatic unconscious (and sometimes biased) ''fast system'' and a ''slow system'', which when triggered may revisit earlier automatic reactions. 
We make several observations from two crowdsourcing experiments of gender bias in coreference resolution, using self-paced reading to study the ''fast'' system, and question answering to study the ''slow'' system under a constrained time setting.
We find that humans and models show comparable biases. 
On real-world data humans make ~3% more gender-biased decisions compared to models, while on synthetic data models are ~12% more biased.

Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

In this work, we use question answering as a general framework for studying how eye movements in reading reflect the reader's goals, how they are pursued, and the extent to which they are achieved. We leverage fine-grained annotations of task-critical textual information to perform a detailed comparison of eye movements in information-seeking and ordinary reading regimes. We further examine how eye movements during information seeking relate to question answering behavior. We find that reading times, saccade patterns and sensitivity to the linguistic properties of the text are all strongly and systematically conditioned on the reading task, and further interact with question answering behavior. The observed reading patterns are consistent with a rational account of cognitive resource allocation during task-based reading.


Eye Movements in Information-Seeking Reading

Normative accounts of decision-making predict that people attempt to balance the immediate rewards associated with correct responses against the costs of deliberation. However, humans frequently deliberate longer than normative models say they should. We propose that people try to optimize not only their rate of material rewards, but also their rate of information gain. A computational model that implements this idea successfully mimics human decision makers, reproducing key patterns of behavior not predicted by alternative models. Moreover, simulations reveal a normative basis for our model: An agent that exchanges even a small amount of immediate reward for information will improve its decision-making ability through learning, allowing it to earn more reward in the long run than an agent disinterested in information. Maximizing a combination of reward and information rate is a simple yet effective strategy for solving the speed-accuracy tradeoff that may resolve lingering mysteries about human decision-making.

When to choose: Information seeking in the speed-accuracy tradeoff

Local coherence is a phenomenon in human sentence processing whereby word sequences within a sentence incur processing difficulty when they have a plausible reading different from their true syntactic structure as disambiguated by the global context. Prior research (Tabor, Galantucci, & Richardson, 2003) indicates that more plausible substrings incur more processing difficulty than less plausible ones. In the current article, we challenge this view by providing evidence from two experiments which show that local semantic plausibility can actually facilitate processing. We additionally test whether syntactic statistics can modulate local coherence effects, a prediction made by Lossy-Context Surprisal (LCS; Futrell, Levy, & Gibson, 2020; Hahn, Futrell, Levy, & Gibson, 2022). Although we do not find evidence for effects of embedding bias, our overall results cannot be fully explained by any existing account of local coherence alone. We discuss implications for theories of sentence processing.

How Do Syntactic Statistics and Semantic Plausibility Modulate Local Coherence Effects

Context changes expectations about upcoming words—following a story involving an anthropomorphic peanut, comprehenders expect the sentence the peanut was in love more than the peanut was salted, as indexed by N400 amplitude (Nieuwland & van Berkum, 2006). This updating of expectations has been explained using Situation Models—mental representations of a described event. However, recent work showing that N400 amplitude is predictable from distributional information alone raises the question whether situation models are necessary for these contextual effects. We model the results of Nieuwland and van Berkum (2006) using six computational language models and three sets of word vectors, none of which have explicit situation models or semantic grounding. We find that a subset of these can fully model the effect found by Nieuwland and van Berkum (2006). Thus, at least some processing effects normally explained through situation models may not in fact require explicit situation models.


Can Peanuts Fall in Love with Distributional Semantics?

We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on the spoken language comprehension. We compare the ERP signature (N400) associated to each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (i.e., measures of how predictable words in their linguistic context) generated on the basis of different types of language models (n-gram and transformer models) predict N400 responses for each word. We find that cognitive effort (measured by surprisal) in a multimodal setting significantly differs from that in a unimodal setting. The analysis of surprisal estimates from language models reveals that while Transformer-based models provide better fit to N400 signals in audio-only setting, 2-gram language models fit better in the multimodal setting, suggesting that local verbal context significantly influences cognitive processing effort in a multimodal setting.


Are words equally surprising in audio and audio-visual comprehension?

To investigate how co-speech gestures modulate linguistic understanding, we experimentally test two Bayesian Pragmatic models. We identify the semantic effect of a spoken or gestural utterance with the change it makes in listener’s probabilistic predictions of the speaker’s communicative intentions. We focus on action-expressing gestures and the respective verbs that correspond to action-affording instruments or, respectively, their denoting nouns. Combining Pustejovsky’s’ Generative Lexicon approach with Gibson’s affordance theory, we ask: (1) Does a co-speech gesture make any difference for semantic comprehension and the corresponding probabilistic prediction? (2) Is the semantic effect of a gesture similar or identical to the one of the corresponding verb? (3) To which extent does the gesture’s semantic effect depend on the listener’s recognition of the gesture as an expression of the corresponding verb? (4) Does the comprehended affordance predict the instrument better than the co-occurrence statistics (GloVe) regarding the verb and the noun?


Modelling the Integration of Co-Speech Gestures into Sentence Meaning Composition

Moral learners face an underdetermination problem – the rules they are taught cannot account for all novel cases. One way learners solve this problem is through closure principles, through which a learner assumes that anything that isn’t explicitly forbidden is permitted, and vice versa. The current work aims to explore whether closure rules are used when reasoning about obligations. Building on previous work, we ask 1) whether adults and children use obligation rules in a similar manner to rules about permissibility and impermissibility, and 2) how early in life these inference abilities emerge. Across two studies, we explore inferences about obligation from both adults (N = 120, Mage = 33.73 years) and children (N = 103, Mage = 5.52 years). We found that while both adults and children rationally learn closure principles consistent with deontic logic, children and adults make opposite inferences about novel cases when provided with obligation rules.


Underdetermination and Obligation Rules: Adult and Children’s use of Closure Principles in Moral Learning

A growing body of literature has revealed ordinary causal judgement to be sensitive to normative factors, such that a norm violating agent is regarded more causal than their non-norm violating counterpart. The present paper aims to tease apart two competing explanations: the Responsibility View and the Bias View. The Bias View, but not the Responsibility View, predicts features peripheral to the agent’s responsibility to impact causal attributions. In a series of three preregistered experiments (N = 1162), we present new evidence that the Norm Effect arises from such peripheral features, namely from nonpertinent or entirely silly norm violations. Further, we show that this effect cannot be explained by recourse to the agent’s foreknowledge or desire of the outcome, nor by its foreseeability: the Norm Effect arises even when participants judge the norm violating agent’s doing as equally foreseeable. This, we argue, provides evidence in favour of the Bias View.


Downloads

Next from CogSci 2023

Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from CogSci 2023

Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

Downloads