United States

People can understand images that vary in visual abstraction---from detailed illustrations to schematic icons. 
To what degree are current vision algorithms sensitive to such variation when encoding their meaning? 
We first obtained &gt;80K human-generated sketches produced under different time limits (4s, 8s, 16s, 32s; N=5,563 participants) and AI-generated sketches (Vinker et al., 2022) produced under different ink limits (8, 16, 32 strokes) of 2,048 real-world objects spanning 128 categories from the THINGS dataset (Hebart et al.,2019).
We then evaluated how 13 state-of-the-art vision algorithms varying in architecture and training represented the meaning of these sketches, both at the category and exemplar levels, and without additional training.
We found that these models were generally sensitive to variation in visual abstraction, although some models were substantially more sensitive than others.
Together, these findings suggest promising avenues for building better models of human visual abstraction.

CogSci 2023

Evaluating machine comprehension of sketch meaning at different levels of abstraction

People can understand images that vary in visual abstraction---from detailed illustrations to schematic icons. 
To what degree are current vision algorithms sensitive to such variation when encoding their meaning? 
We first obtained >80K human-generated sketches produced under different time limits (4s, 8s, 16s, 32s; N=5,563 participants) and AI-generated sketches (Vinker et al., 2022) produced under different ink limits (8, 16, 32 strokes) of 2,048 real-world objects spanning 128 categories from the THINGS dataset (Hebart et al.,2019).
We then evaluated how 13 state-of-the-art vision algorithms varying in architecture and training represented the meaning of these sketches, both at the category and exemplar levels, and without additional training.
We found that these models were generally sensitive to variation in visual abstraction, although some models were substantially more sensitive than others.
Together, these findings suggest promising avenues for building better models of human visual abstraction.

poster

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Although object detection AI plays an important role in many critical systems, corresponding Explainable AI (XAI) methods remain very limited. Here we first developed FullGrad-CAM and FullGrad-CAM++ by extending traditional gradient-based methods to generate object-specific explanations with higher plausibility. Since human attention may reflect features more in-terpretable to humans, we explored the possibility to use it as guidance to learn how to combine the explanatory information in the detector model to best present as an XAI saliency map that is interpretable (plausible) to humans. Interestingly, we found that human attention maps had higher faithfulness for explaining the detector model than existing saliency-based XAI methods. By using trainable activation functions and smoothing kernels to maximize the XAI saliency map similarity to human attention maps, the generated map had higher faithfulness and plausibility than both existing XAI methods and human atten-tion maps. The learned functions were model-specific, well generalizable to other databases.

Human Attention-Guided Explainable AI for Object Detection

For the last decade, convolutional neural networks (CNNs) have vastly superseded their predecessors in nearly all vision tasks in artificial intelligence, including object recognition. However, despite abundant advancements, they continue to pale in comparison to biological vision. This chasm has prompted the development of biologically-inspired models that have attempted to mimic the human visual system, primarily at a neural level, which is evaluated using standard dataset benchmarks. However, more work is needed to understand how these models perceive the visual world. This article proposes a state-of-the-art procedure that generates a new metric, Psychophysical-Score, which is grounded in visual psychophysics and is capable of reliably estimating perceptual responses across numerous models --- representing a large range in complexity and biological inspiration. We perform the procedure on twelve models that vary in degree of biological inspiration and complexity, we compare the results against the aggregated results of 2,390 Amazon Mechanical Turk workers who together provided $\sim2.7$ million perceptual responses. Each model's Psychophysical-Score is compared against the state-of-the-art neural activity-based metric, Brain-Score. Our study indicates that models with a high correlation to human perceptual behavior also have a high correlation with the corresponding neural activity.

Psychophysical-Score: A Behavioral Measure for Assessing the Biological Plausibility of Visual Recognition Models

Relational ability—the ability to compare situations or ideas and discover common relations – is a key process in higher-order cognition that underlies transfer in learning and creative problem solving. For this reason, it has generated intense interest both among developmentalist and in cross-species comparative studies. The gold standard for evaluating relational ability is the Relational-Match-to-Sample (RMTS) task (Premack, 1983). Current work in cognitive development has produced inconsistent results as to when children are able to pass the RMTS, with Christie and Gentner (2014) finding earlier success than do Hochmann et al. (2017). In this research, we attempt to resolve this issue. We first describe two studies that bear out and extend the Christie and Gentner (2014) findings. We then discuss factors that might explain the discrepancy between the findings.

When do Children Pass the Relational-Match-To-Sample Task?

Abstract reasoning in early childhood is often described as following a "relational shift," over which children become increasingly sensitive to relations. However, recent work has challenged the generality of this account, showing that children in the US and China follow distinct trajectories in a relational match-to-sample task (Carstensen et al., 2019). This difference aligns with multiple cultural and linguistic factors implicated in relational reasoning, in which English speakers in the US and Mandarin speakers in China appear at opposite ends of a continuum spanning from a focus on objects (US) to relations (CN). We explore early relational reasoning in a context that represents a cultural middle ground with a key linguistic similarity (noun spurts) to the US: Korean-learning children in South Korea. In two experiments with 262 Korean children, we document relational reasoning in this novel cultural context, revealing similarities and differences to developmental trajectories in the US and China.

Relational abstraction in early childhood: Three cultures and three trajectories

Analogy is considered a core cognitive capacity that enables relational reasoning. It has been commonly believed that relational reasoning can be tested by the four-term analogy task of type “what is to c as b is to a?” or “a:b::c:?” in short. In this study, we challenge this common belief by investigating an alternative hypothesis that this classic task is solvable only by attributional reasoning on attributional similarities, rather than relational reasoning on the relation of the two relations. A word vector model was employed to demonstrate how these two types of reasoning were performed in the classic three-stem analogy task “a:b::c:?” and the two-stem analogy task “a:b::?:?”. The results showed that the two-stem analogy task was solvable only using relational reasoning, while the three-stem analogy task was solvable using both types of reasoning. It suggests that the two-stem analogy task is a better test for relational reasoning.

Two-stem Analogy Task is a Better Test for Relational Reasoning

Analogical reasoning is central to learning and thought with a key focus on similarity in which the same system of relations holds across different sets of elements. The current research investigated both first-order relation and second-order relation, represented in the combination pattern of symbols following the syntactic rules. The primary goal was to specify the neural substrates for abstract syntactic rule analogy.Using fMRI, the study identifies brain regions involved in this task, including left rostrolateral prefrontal cortex (rlPFC), left dorsolateral prefrontal cortex (dlPFC), and right precuneus. The results of this study reveal that although analogical reasoning of syntactic rules recruits similar brain regions as those in visuospatial and semantic analogy, their activation patterns are not the same. The role of rlPFC, the core area for analogical reasoning, needs to be reconsidered, and the complexity of precuneus is worth more attention.

Neural Correlates of Analogical Reasoning of Syntactic Rules

Detecting covariation in sequential events provides us with a powerful means of inferring the causal structure of our world. However, people often overestimate the causal relationship between unrelated events, a phenomenon referred to as illusory causation. This tendency is greatest when the putative effect occurs frequently; the widely replicated outcome density (OD) effect. Most laboratory research on illusory causation and the OD effect has focused on possible causes of a positive outcome, such as a drug that causes patient recovery. Despite its relevance, relatively few studies have examined illusory causation in cases where a cue is hypothesized to generate an unfavorable (negative) outcome, such as a drug that produces unwanted side effects. Here, we directly compared how people develop illusory beliefs about the generation of positive versus negative outcomes. We presented all participants with a drug treatment that was hypothesized to cause high readings of a fictitious cell count (but had no effect on cell count across a series of learning trials). We manipulated whether a high cell count occurred frequently or infrequently and whether a high cell count should be considered a beneficial medical outcome or an undesirable side effect. We found consistent evidence of an OD effect but no effect of the valence of the high cell count outcome. This suggests that illusory beliefs are not controlled by the desirability of the cause-effect relationship. We discuss implications for theories of applied causal reasoning.

Learning about Benefits and Side Effects of a Bogus Treatment are Similarly Influenced by the Frequency of the Outcome Occurring

Most research on causal judgment operates on an implicit premise: that the only relevant candidates for actual causation are individual variables, corresponding to the distinct nodes of a causal system. Instead, we claim that causal judgment can be about plural causes featuring more than one variable. We show that plural cause judgments are influenced by the prior probabilities of events just like other actual causation judgments. We further show that this dependence on probabilities cannot be explained away by supposing that subjects estimate the strength of a plural cause by combining their estimates for the individual variables that feature in it. Instead, plural causes appear to be apprehended holistically, as the object of an ad hoc computation, providing evidence that the notion of plural cause is a category of human judgment in its own right.

Plural causes in causal judgment

Predictive coding theories argue that recent experience establishes expectations that generate prediction errors when violated. Neural correlates of prediction errors have been widely reported, but their perceptual consequences remain unknown. We had observers perform a dual-report task on the orientation of briefly presented gratings within predictable or random sequences. Observers first made a speeded response to categorize grating orientation (clockwise or counterclockwise from vertical), then reproduced the orientation without time pressure by rotating a bar. This allowed us to separately assess response speed and accuracy for the same stimulus. Critically, on half the trials, the target orientation was unexpected, due to the spatiotemporal structure established by the preceding gratings. Observers responded faster and more accurately to gratings that were unexpected. These findings reveal the behavioural consequences of prediction error and indicate that unexpected events are prioritized by the visual system both in terms of processing speed and representational fidelity.

Unexpected visual events elicit faster and more accurate behavioural responses

Seeking plausible models for brain computation has been a continuing effort in brain encoding and decoding. Most prior works have mapped the association between stimulus representation from language models and fMRI brain activity using ridge regression. However, these models are not biologically plausible from the perspective of representing neural dynamics of the brain underlying the fMRI recordings. In this work, our primary motivation is to challenge ridge regression models with simple neural architectures such as echo state network (ESNs) and long short-term memory (LSTMs) on the brain encoding task that requires full-sentence processing in the task of reading short sentences. We explore various pre-trained Transformer language models for computing sentence representations and predict the fMRI brain activity from simple neural architectures that include initial layers with random parameterization and that do not require explicit training. Experiment results show that (i) ESNs with online learning can accurately predict the fMRI brain activity comparable to ridge regression models, (ii) Both cell-state (internal memory representation related to long term memory) and out-gate (related to short term memory) of LSTM display an equal level performance during short sentences in random LSTMs, (iii) left hemisphere language area has higher predictive brain activity versus right hemisphere language area, (iv) ESNs with online learning yield superior performance over offline learning, indicating the biological plausibility of ESNs and the cognitive process of sentence reading, and (v) among all the variants of transformer models, Longformer features facilitate better accuracy when utilized with both ridge regression and ESN online learning models. The proposed framework that combines input featurization, dynamic memory and learning modules offers a flexible, biologically plausible architecture for investigating brain encoding in neuroscience.

Premium content

Evaluating machine comprehension of sketch meaning at different levels of abstraction

Next from CogSci 2023

Human Attention-Guided Explainable AI for Object Detection