United States

Previous computational models of early language acquisition have shown how linguistic structure of speech can be acquired using auditory or audiovisual learning mechanisms. However, real infants have sustained access to both uni- and multimodal sensory experiences. Therefore, it is of interest how the uni- and multimodal learning mechanisms could operate in concert, and how their interplay might affect the acquisition dynamics of different linguistic representations. This paper explores these questions with a computational model capable of simultaneous auditory and audiovisual learning from speech and images. We study how the model’s latent representations reflect phonemic, lexical, and semantic knowledge as a function of language experience. We also test how the findings vary with differential emphasis on the two learning mechanisms. As a result, we find phonemic learning always preceding lexical learning, followed by semantics. This pattern also emerges when the learner is all the time prioritizing audiovisual associative learning instead of lexical or sublexical representations. The result illustrates how the acquisition order of linguistic representations is decoupled from the primary learning objectives (mechanisms) of the learner, and how the emergence of phonemes and words can be facilitated by both auditory and audiovisual learning in a synergetic manner.


CogSci 2023

Computational Insights to Acquisition of Phonemes, Words, and Word Meanings in Early Language: Sequential or Parallel Acquisition?

technical paper

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Representational gestures are co-speech gestures that carry semantic content related to the content of speech. Previous studies focusing on adult-adult conversation have investigated the temporal alignment of gestures and speech finding that the overwhelming majority of representational gestures are produced right before the lexical content they refer to (their lexical affiliate, LA). However, nothing is yet known about whether caregivers would also time their gestures in the same way in naturalistic interactions. We annotated representational gestures from a large corpus (anonymized) of semi-naturalistic conversations between caregivers and their 3-4 year old children (n = 899 gestures from n=36 caregivers). We found that, just as in adult-directed language (ADL), representational gestures in child-directed language (CDL) were more tightly linked to the onset of LAs than the onset of the utterance in which Las were produced (hence planned when full events are encoded); with the overall majority of the representational gestures starting before their LAs. We further found that age of acquisition (AoA) rating of the LA had a significant effect on the speech-gesture latency. We found that for words acquired earlier, the gesture’s stroke (the meaningful part of a gesture) tended to be produced before the LA’s onset; for the later acquired word, the stroke tended to be produced at the same time or after the onset of the LA in speech. Our findings suggest that: (1) Regardless of their addressee, speakers always time the production of representational gestures to specific conceptual/linguistic units, rather than the full event/utterance. (2) In contrast to ADL, caregivers’ gestures may support addresses’ linguistic processing not only by supporting word prediction (of likely better known words), but also by supporting the learning of conceptual features (of likely less well-known words).


How Speech and Representational Gestures Align in Child-Directed Language: a Corpus-based Study

While naturalistic daylong audio recordings of children’s auditory environments have the potential to reveal key insights about the input children receive and inform our theories of language development, it also presents various methodological hurdles. In the present work, we used three fully transcribed daylong audio recordings to investigate the challenge of manually extrapolating aggregate statistics and quantify the kinds of sampling choices daylong researchers can make. Our findings highlight sampling choices that maximize sampling from the full distribution of the day and potential tradeoffs between human effort and obtaining accuracy.


Optimizing Random Sampling of Daylong Audio

Using evidential expressions to indicate one’s source of information for an utterance tends to convey uncertainty on the speaker’s part. Previous accounts of this uncertainty inference attribute it to either extralinguistic reasoning about evidence directness, or to pragmatic reasoning about alternative utterances. Here we present a novel hybrid account, and introduce a set of utterances which allows us to tease apart the three accounts’ predictions. We test these predictions in two studies by manipulating the directness of evidence indicated by an evidential expression. Exp. 1 shows that listeners infer more uncertainty with extreme values of directness. Exp. 2 shows that speakers are more likely to indicate evidence in contexts where the evidence is unreliable. We argue that these findings support an account which involves both extralinguistic and pragmatic reasoning, and develop a formal implementation of such an account within the Rational Speech Act framework.


Evidential uncertainty involves both pragmatic and extralinguistic reasoning: a computational account

Projection inferences are inferences about speaker commitment to a content embedded under an entailment-canceling operator, such as polar interrogatives with clause-embedding predicates (e.g., Does John know that Julian dances salsa?) Inferred speaker belief regarding whether Julian dances salsa is modulated by multiple factors, including the predicate used, interlocutors' prior beliefs about the content, and the content's at-issueness. We propose a first step towards a unified probabilistic model of projection inferences about content embedded under clause-embedding predicates, couched in the RSA framework. In a behavioral study, we investigate the beliefs about contents embedded under the predicates ``think'' and ``know'' that listeners ascribe to the speaker and a potential attitude holder (John). The empirical data were used to inform the model. The resulting model predictions mirror some, but not all, of the qualitative empirical patterns. This is the first step toward a systematic analysis of projection inferences using probabilistic pragmatic models.


Towards a computational account of projection inferences in clause-embedding predicates

When we reason about what could have been, some possibilities seem plausible, and others far-fetched. According to a recent theory, counterfactual possibilities are plausible if they can be generated by making local, probabilistic adjustments to the causes of what actually happened. We provide evidence that people think about counterfactuals in this way even when they have to infer the causes of what happened. We told participants about the diet of a fictional animal, and then asked them simple counterfactual questions. For example, given that the animal has eaten 1 berry today, how much food could it plausibly have eaten instead? When the amount of food eaten by the animal licensed an inference about a causally upstream variable, participants inferred the state of this variable and used it to guide their counterfactual plausibility judgments. 
More generally, the distribution over counterfactual values derived from participants' judgments was remarkably similar to the distribution predicted by the model.


Causal inference shapes counterfactual plausibility

There is increasing pressure on social media companies to reduce the spread of misinformation on their platforms. However, they would prefer not to be the arbiters of truth as the truth can be subjective or otherwise hard to determine. Instead, they would prefer that social media users themselves show better discernment when deciding which information to share. Here we show that allowing people to share only those social media posts that they have indicated are true significantly improves sharing discernment, as measured by the difference in the probability of sharing true information versus the probability of sharing false information. Because it doesn’t require social media companies to be the arbiters of truth, this self-censorship intervention can be employed in situations where social media companies suspect that individuals are propagating misinformation but are not sufficiently confident in their suspicions to directly censor the individuals involved. As such, self-censorship can usefully supplement externally imposed (i.e. traditional) censorship in reducing the propagation of false information on social media platforms.


Self-Censorship Appears to be an Effective Way of Reducing the Spread of Misinformation on Social Media

Pre record assistants have become fixtures in everyday settings, but most research focuses on their development rather than their use following deployment. To facilitate study of their use in office settings, we introduce OfficeDial, a multimodal dataset containing audio recordings, transcriptions, eye tracking data, and screen recordings from conversations between humans and Pre record assistants in office environments. Conversations are paired with physical and behavioral measures of cognitive load. We study the associations between verbal behavior and noise level and reveal key relationships between verbal redundancy, disfluency, and noise level. We make our new dataset available to interested researchers to inspire further exploration.


Linguistic Cognitive Load Analysis on Dialogues with an Intelligent Pre record Assistant

Not too long ago most headlines talked about our fear of AI. Today, AI is ubiquitous, and the conversation has moved on from whether we should use AI to how we can trust the AI systems that we use in our daily lives. In this talk I look at some key technical ingredients that help us build confidence and trust in using intelligent technology. I argue that intuitiveness, interaction, explainability and inclusion of human domain knowledge are essential in building this trust. I present some of the techniques and methods we are building for making AI systems that think and interact with humans in more intuitive and personalised ways, enabling humans to better understand the solutions produced by machines, and enabling machines to incorporate human domain knowledge in their reasoning and learning processes.

Premium content

Downloads

Next from CogSci 2023

How Speech and Representational Gestures Align in Child-Directed Language: a Corpus-based Study