Netherlands

In hierarchical perception-action (PA) learning, agents discover invariants between percepts and actions that are structured hierarchically, from very basic immediate links to higher-level, more abstract notions. In practice, existing work tends to either focus on the general theory at the expense of details of the proposed mechanisms, or specify a-priori the contents of some layers. Here, we introduce a framework that does without such constraints. We demonstrate the framework in a simple 2D environment using an agent that has minimal perceptual and action abilities. We vary the perceptual abilities of the agent to explore how the specifics of this aspect of the agent&#39;s body might affect PA learning and find unexpected consequences. The contribution of this paper is therefore twofold, (1) we add a novel framework to the literature on PA learning, using, in particular curiosity-based reinforcement learning (RL) to implement the necessary learning mechanisms, and (2) we demonstrate that even for very simple agents, the relation between the specifics of an agent&#39;s body and its cognitive abilities is not straightforward.

**Authors:**

Tara Carette: Radboud University; Serge Thill: Radboud University

CogSci 2024

A general framework for hierarchical perception-action learning

We are pleased to announce the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), which will be held in Vancouver, British Columbia at the Vancouver Convention Centre – West Building from 20-27 February, 2024.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-24 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

We expect for AAAI-24 to be an in-person conference – one author of all accepted papers will be expected to present work in person unless there are exceptional circumstances that prevent this.

In order to access the AAAI-24 event page you need to register [here](https://aaai.org/aaai-conference/registration/)

AAAI 2024

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. 

In hierarchical perception-action (PA) learning, agents discover invariants between percepts and actions that are structured hierarchically, from very basic immediate links to higher-level, more abstract notions. In practice, existing work tends to either focus on the general theory at the expense of details of the proposed mechanisms, or specify a-priori the contents of some layers. Here, we introduce a framework that does without such constraints. We demonstrate the framework in a simple 2D environment using an agent that has minimal perceptual and action abilities. We vary the perceptual abilities of the agent to explore how the specifics of this aspect of the agent's body might affect PA learning and find unexpected consequences. The contribution of this paper is therefore twofold, (1) we add a novel framework to the literature on PA learning, using, in particular curiosity-based reinforcement learning (RL) to implement the necessary learning mechanisms, and (2) we demonstrate that even for very simple agents, the relation between the specifics of an agent's body and its cognitive abilities is not straightforward.

**Authors:**

Tara Carette: Radboud University; Serge Thill: Radboud University

poster

The 46th Annual Meeting of the Cognitive Science Society was an in-person meeting held in Rotterdam, The Netherlands at the Postillion Hotel & Conference Centre.

**ON-DEMAND PROGRAM ACCESS AFTER THE CONFERENCE** 
Recordings of the invited program are now available. You can access them by clicking on the 'Schedule' icon on the left. Select view by week and navigate to the day and time slot of the recording you wish to view. Click on the time slot to access the recording.

Recordings available (click on each recording to view):

**Thursday, July 25** 
0900: [Keynote Speaker Morgan Barense on Enhancing real-world event memory](https://underline.io/events/465/sessions/17997/lecture/99084-dynamics-between-minds-and-the-environment) 
1000: [Gleitman Award Winner Isabelle Dautriche on Language Foundations: Insights from acquisition, communication, cognition, and more](https://underline.io/events/465/sessions/18000/lecture/99085-gleitman-talk) 
1415: [Invited Symposium: Dynamics between minds and the environment](https://underline.io/events/465/sessions?eventSessionId=18016&searchGroup=lecture) 
1700: [Rumelhart Prize Presentation Speaker Alison Gopnik on Exploit, explore, empower: Three ages and three intelligences](https://underline.io/events/465/sessions/18028/lecture/99161-rp1-rumelhart-prize-presentation) 

**Friday, July 26** 
0900: [C.L. de Carvalho-Heineken Prize Keynote Speaker Kia Nobre on Focusing in memory](https://underline.io/events/465/sessions/18035/lecture/99162-hp1-c-l-de-carvalho-heineken-prize-keynote-address) 
1030: [Rumelhart Symposium: Childhood as exploration](https://underline.io/events/465/sessions/18038/lecture/100346-childhood-as-exploration) 
1415: [Elman Prize Symposium](https://underline.io/events/465/sessions?eventSessionId=18053&searchGroup=lecture) 
1600: [Invited Symposium: Dynamics between minds](https://underline.io/events/465/sessions?eventSessionId=18065&searchGroup=lecture) 
1745: [Keynote Speaker Andrea E. Martin on Neural dynamics encode the structure and statistics of language](https://underline.io/events/465/sessions/18077/lecture/99271-neural-dynamics-encode-the-structure-and-statistics-of-language) 

**Saturday, July 27** 
0900: [Keynote Speaker Gregor Schöner on How higher cognition emerges from the dynamics of strongly interacting neural populations](https://underline.io/events/465/sessions/18083/lecture/99273-dynamics-within-the-mind) 
1000: [Glushko Talks](https://underline.io/events/465/sessions?eventSessionId=18086&searchGroup=lecture) 
1500: [Invited Symposium: Dynamics within minds](https://underline.io/events/465/sessions?eventSessionId=18100&searchGroup=lecture) 


#### **CogSci 2024 Program Book**
<div style="position:relative;padding-top:0;width:900px;height:500px;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/cble/" seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


**The Program Book can be downloaded [here](https://drive.google.com/file/d/1yRVp1cBqRnAbdYi4GXWV_pZeyWNsRquv/view?usp=sharing).**

#### **Letter from the President**

Welcome to the 2024 Meeting of the Cognitive Science Society in Rotterdam! CogSci 2024 brings together a large community of cognitive scientists who have traveled here from around the world, as well as a smaller group of remote presenters. I would like to extend an especially warm welcome to our first-time CogSci meeting attendees, and hope that 
their experience inspires them to join our future meetings for many years to come.

I want to thank this year’s conference co-chairs, Larissa K Samuelson, Stefan Frank, Mariya Toneva, Allyson Mackey, and Eliot Hazeltine, for putting together an amazing series of invited talks and symposia around the theme of Dynamics of Cognition. The co-chairs also deserve credit for handling a record-breaking number of submissions and producing a truly exciting conference program 

Beyond the invited talks and symposia dedicated to this year’s conference theme, I would like to highlight the presentations and events honoring the 
Rumelhart, Elman, Gleitman and C .L . de Carvalho Heineken prize winners, including the Rumelhart reception on Thursday. As you may know, we are recording all of the invited talks and symposia, and all prize-winners’ talks, and will be making this content available after the conference. 

The Cognitive Science Society relies on the volunteer efforts of our Governing Board members who work on matters related to our membership; conference policies; diversity and inclusion, international, and outreach initiatives; prizes and much more. We welcome interest from Society members who would like to become more involved in what we do. The 
Society is deeply grateful to Rick Dale and Andrea Bender, who serve as editors of the two Society journals, Cognitive Science and Topics in Cognitive Science respectively. We especially acknowledge our editors’ generosity in donating their stipends each year to D&I and outreach efforts. 

This year’s Annual Meeting depends critically on the support of Marischal De Armond and his team at Podium Conference Specialists: Sharon Zwack, Cendrine De Vis, Sarah-Kate Burke, and Rachel Waller worked tirelessly to secure the venue and handled a multitude of logistical and organizational issues, from arranging coffee breaks and receptions to making sure that a huge number of posters fits comfortably into the available space. Finally, and importantly, I would like to acknowledge the Cognitive Science Society Executive Officer, Erica Wojcik, for managing the complex and ever expanding sphere of our Society’s activities.

I hope you enjoy this year’s meeting and the many different opportunities to engage with the vibrant community of cognitive scientists gathered here. I also hope you get a chance to explore the great city of Rotterdam with its many attractions, museums, restaurants and scenic views. If you’d like to talk more about our Society, or simply want to say hello, please stop by!

![](https://assets.underline.io/markdown_image/1/image/3c66d0e21fff30e204df1138ffd15173.jpeg)

**Anna Papafragou** 
President, Cognitive Science 
Society 2023-2024

**CODE OF CONDUCT** 
By attending the CogSci 2024 Conference, you are required to adhere to the society’s [Code of Conduct](https://drive.google.com/file/d/16-6KkptF0Gn3ZYGJDlpqpwTdImw45Ng0/view?usp=drive_link).

**ABOUT THE COGNITIVE SCIENCE SOCIETY** 
The Cognitive Science Society brings together researchers from around the world who hold a common goal: understanding the nature of the human mind. The mission of the Society is to promote Cognitive Science as a discipline, and to foster scientific interchange among researchers in various areas of study, including Artificial Intelligence, Linguistics, Anthropology, Psychology, Neuroscience, Philosophy, and Education.

The Society is a non-profit professional organization and its activities include sponsoring an annual conference and publishing the journals Cognitive Science and TopiCS.

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

It looks like you are not registered for this event. 

To access the site please register [**here**](https://cognitivesciencesociety.org/registration/). 

Please register!

The 46th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Compositionality is considered a key hallmark of human language. However the degree to which people utilize compositionality in practice has not been directly quantified and recent studies showing re-use of learned linguistic “chunks” suggest that compositionality may be employed less frequently than commonly believed. Here, we aim to assess the degree of compositionality in English, focusing on the case of adjective-noun combinations as a testbed for our approach. Using corpus analysis, large language models (LLMs), and newly collected human acceptability ratings, we find that: (1) English vastly underutilizes the compositional potential of adjective–noun combinations; and (2) LLMs struggle to predict human acceptability ratings of unattested adjective-noun pairs, exposing an important shortcoming of these state-of-the-art models. Taken together, our findings shed new light on the role of compositionality in language and suggest a new challenging benchmark for further improving LLMs.

**Authors:**

Aalok Sathe: Massachusetts Institute of Technology; Evelina Fedorenko: Massachusetts Institute of Technology; Noga Zaslavsky: UCI

Compositionality is underutilized in language: The case of English adjective-noun phrases in humans and large language models

The space of human goals is tremendously vast; and yet, from just a few moments of watching a scene or reading a story, we seem to spontaneously infer a range of plausible motivations for the people and characters involved. What explains this remarkable capacity for intuiting other agents' goals, despite the infinitude of ends they might pursue? And how does this cohere with our understanding of other people as approximately rational agents? In this paper, we introduce a sequential Monte Carlo model of open-ended goal inference, which combines top-down Bayesian inverse planning with bottom-up sampling based on the statistics of co-occurring subgoals. By proposing goal hypotheses related to the subgoals achieved by an agent, our model rapidly generates plausible goals without exhaustive search, then filters out goals that would be irrational given the actions taken so far.

We validate this model in a goal inference task called Block Words, where participants try to guess the word that someone is stacking out of lettered blocks. In comparison to both heuristic bottom-up guessing and exact Bayesian inference over hundreds of goals, our model better predicts the mean, variance, efficiency, and resource rationality of human goal inferences, achieving similar accuracy to the exact model at a fraction of the cognitive cost, while also explaining garden-path effects that arise from misleading bottom-up cues. Our experiments thus highlight the importance of uniting top-down and bottom-up models for explaining the speed, accuracy, and generality of human theory-of-mind.

**Authors:**

Tan Zhi-Xuan: Massachusetts Institute of Technology ; Gloria Kang: Massachusetts Institute of Technology; Vikash Mansinghka: MIT; Josh Tenenbaum: MIT

Infinite Ends from Finite Samples: Open-Ended Goal Inference as Top-Down Bayesian Filtering of Bottom-Up Proposals

The commonsense question answering (CQA) system is a type of conversational system that can generate answers to commonsense questions posed by human users. Most current research on commonsense question-answering has focused on proposing different techniques in natural language processing and text information retrieval. However, for humans, retrieving and organizing desired answers from text knowledge related to commonsense questions is far less intuitive and comprehensive than it is when using multi-modal knowledge, such as related images and videos. Motivated by this, our paper proposes a framework for trying the acquisition of diverse modal information from various perspectives, and embedding and integrating it into CQA tasks, further improving the performance and user experience. Specifically, this paper proposes the integration of multi-modal knowledge, including images retrieved with questions, image description statements, image scene graphs, and knowledge sub-graphs retrieved with questions, into a CQA system. It introduces a parallel embedding technique for this multi-modal knowledge and employs an alignment-interaction-fusion mechanism to facilitate the seamless integration of this multi-modal knowledge. Through extensive and rigorous experiments, the effectiveness and superiority of our proposed method are demonstrated, achieving a notable 4.0% and 3.4% improvement in accuracy on the OpenBookQA and CommonsenseQA datasets, respectively.

**Authors:**

Yongqiang Zhao: Peking University; Zhi Jin: Peking University; Feng Zhang: Academy of Military Sciences; Xinhai Xu: Academy of Military Sciences; Donghong Liu: Academy of Military Sciences

Bridging the Gap: Advancing Commonsense Question Answering with Integrated Multi-Modal Knowledge

People’s color-concept associations influence many processes underlying visual cognition from object recognition to information visualization interpretation. Thus, a key goal in cognitive science is developing efficient methods for estimating color-concept association distributions over color space to model these processes. Here, we investigated the ability of GPT-4, a multimodal large language model, to estimate human-like color-concept associations. We collected human association ratings between 70 concepts spanning abstractness and 71 colors spanning perceptual color space and compared these ratings to analogous ratings from GPT-4, when it was given concepts as words and colors as hexadecimal codes. GPT-4 ratings were correlated with human ratings, comparably to state-of-the-art image-based methods. Variation in human-GPT rating correlations across concepts was predicted by concept abstractness, but this effect was superseded by specificity (peakiness; inverse entropy) of color-concept association distributions. Our results highlight the viability of using model-generated color-concept association ratings to better understand human color semantics.

**Authors:**

Kushin Mukherjee: University of Wisconsin-Madison; Timothy Rogers: UW-Madison; Karen Schloss: University of Wisconsin - Madison

Estimating human color-concept associations from multimodal language models

We build a computational model of how humans actively infer hidden rules by doing experiments.
The basic principles behind the model is that, even if the rule is deterministic, the learner considers a broader space of fuzzy probabilistic rules, which it represents in natural language, and updates its hypotheses online after each experiment according to approximately Bayesian principles.
In the same framework we also model experiment design according to information-theoretic criteria.
We find that the combination of these three principles---explicit hypotheses, probabilistic rules, and online updates---can explain human performance on a Zendo-style task, and that removing any of these components leaves the model unable to account for the data.

**Authors:**

Top Piriyakulkij: Cornell University; Kevin Ellis Ellis: Cornell

Doing Experiments and Revising Rules\\with Natural Language and Probabilistic Reasoning

Sleep staging holds significant importance in clinical medicine, aiding in the diagnosis of various disorders related to sleep and cognition. However, manually annotating a large amount of sleep data is time-consuming and labor-intensive, making it difficult to achieve. Efficiently utilizing these unannotated data poses a challenging task. We propose a novel self-supervised learning method with Temporal-split Contrastive and Electrode Autoencoder (TsC-EA) for sleep staging. We demonstrate that our method achieves state-of-the-art performance in self-supervised learning on SleepEDF and MASS-SS3. Moreover, experimental results indicate that our method can surpass the performance of supervised learning methods using only 10% of labeled data. Additionally, we explore the application of self-supervised learning in patients with disorder of consciousness. It can assist in diagnosing the severity of DoC through analysis of sleep staging. Staging the sleep patterns of patients with disorders of consciousness can help in diagnosing the severity of their condition.

**Authors:**

Jingcong Li: South China Normal University; Quanlin Chen: South China Normal University; Jiahui Pan: South China Normal University; Haiyun Huang: South China Normal University

A Novel Self-Supervised Learning Method for Sleep Staging and its Pilot Study on Patients with Disorder of Consciousness

Human ecological success stems from our ability to absorb and build upon cultural knowledge, a process we aim to model computationally by integrating individual and cultural learning from language — one of the main vehicles of cultural transmission (e.g. instructions, explanations, stories). In simple video games, our model infers game rules from both interaction data (individual learning) and partial causal models extracted from game descriptions (cultural learning). Given exhaustive descriptions (either hand-written or generated by a model given access to oracle data), models leveraging the two learning sources induce more accurate game rules from limited data than both the individual- and cultural-only controls. Interestingly, descriptions from human game players do not consistently yield better rule induction. We hypothesize that players may preferentially communicate information that will be essential to the others’ future decision-making and we aim to investigate cultural transmission by integrating individual and cultural learning with both causal understanding and decision-making.

**Authors:**

Cédric Colas: MIT; Tracey Mills: MIT; Ben Prystawski: Stanford University; Michael Henry Tessler: Google DeepMind; Noah Goodman: Stanford University; Jacob Andreas: MIT; Josh Tenenbaum: MIT

Naturalistic Transmission of Causal Knowledge between Machines and Humans

We are constantly faced with the question of how to aggregate preferences, views, perspectives and values. This is a problem for individual rational decision makers attempting to trade-off conflicting interests as well as for groups attempting to accommodate individuals with differing needs and interests. The problem of ``value aggregation'' therefore crops up in myriads of places across the social sciences---in rational decision theory, social choice models, and proposals for systems of democratic voting, for instance. These sub-disciplines have formalized proposals for how to deal with value aggregation, though, remarkably, no research has yet directly compared people’s intuitions of two of the most obvious candidates for aggregation--taking the \textit{sum} of all the values (the classic ``Utilitarian'' approach) or the \textit{product} (a less well-known ``contractualist'' approach). In this paper, we systematically explore the proposals suggested by each algorithm. Finally, we compare the judgments of large language models (LLMs) to that of our (human) participants, finding marked differences across model sizes. While the dominant assumptions in fields from decision theory, to AI, to philosophy have favored a utilitarian approach to value aggregation, we find that both humans and performant LLMs prefer a contractualist approach.

**Authors:**

Jared Moore: Stanford University; Sydney Levine: Allen Institute for Artificial Intelligence; Yejin Choi: Allen Institute for Artificial Intelligence

Perceptions of Compromise: Comparing Consqequentialist and Conctractualist Accounts

Self-regulated learning influences students’ learning behaviors and is a significant academic performance factor. Resource management strategy based on self-regulated learning theory is an important indicator for students to demonstrate self-regulatory skills. However, current self-regulated learning and resource management strategy assessments still rely on subjective evaluations and self-assessments, which are time-consuming and laborious. Therefore, we propose a novel method combined with mobile sensing by collecting detailed learning strategy subscales and objective mobile sensing data from N = 211 college students to explore a new approach to assessing self-regulated learning and resource management strategy. To our knowledge, we are the first to propose a mobile sensing approach for assessing self-regulated learning and learning strategies. The method studies the associations between the learning strategy subscales and these daily behavior patterns and presents a set of features for behavior patterns from mobile sensing data. These features represent the self-regulated learning skills of college students.

**Authors:**

Tongyu Zhao: University of Minho; Jiaying Gao: School of Artificial intenlligence; Yu Feng: University of Minho; Yatong Zu: Jilin University; Adriano Tavares: University of Minho; Tiago Gomes: University of Minho; Hao Xu: Jilin University

Behavioral Sensing: An Exploratory Study to Assess Self-Regulated Learning and Resource Management strategy of University Students using Mobile Sensing

Electroencephalography (EEG) and function Near Infrared Spectroscopy (fNIRS), both portable and non-invasive, enhance brain-computer interface (BCI) performance by integrating their spatial and temporal benefits when combined together. However, the fusion of these two signals still faces challenges. To fully unitize the complementarity of EEG and fNIRS for improved performance in EEG-fNIRS BCI, we propose an EEG-fNIRS fusion network based on end-to-end mutual information learning, named EFMLNet. In the model, EEG and fNIRS data are fed into their respective feature extractors for the extraction of temporal and spatial information. Furthermore, their complementary information is fused by two parallel mutual learning modules. Classification experiments were conducted on two public BCI datasets, including motor imagery (MI) and mental arithmetic (MA) tasks. The cross-subject classification accuracy achieved on the MI and
MA datasets was 71.52% and 69.16%, respectively, surpassing the performance of most existing fusion methods and showing promise for real-time hybrid BCI systems.

**Authors:**

Lina Qiu: South China Normal University; Weisen Feng: South China Normal University; Zuorui Ying: South China Normal University; Jiahui Pan: South China Normal University

Premium content

Downloads

Next from CogSci 2024

Compositionality is underutilized in language: The case of English adjective-noun phrases in humans and large language models

Similar lecture

Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations