United States

Understanding a sequence of images as a visual narrative is challenging because it requires not only the understanding of what is shown at a particular moment but also what has changed, been omitted or is out of frame. The human cognitive system makes inferences about the state of the world based on transitions between sequential frames. In this paper, we present a principled analysis of the stylistic differences between two dominant styles of multi-modal narratives, western comics and manga. These two styles differ in terms of screening, ballooning, layout, language, and reading order. We first provide a systematic account of these differences based on an annotated dataset consisting of both comics and manga. We then annotate these datasets with a new feature set and evaluate the contributions of these features through development of a computational model of multi-modal comprehension. The model evaluation is presented through the cloze test that measures the accuracy of the model in predicting unseen next frames given the prior frames in a sequence. Our results provide initial benchmarks and insight into the fundamental challenges that the multi-modal narrative understanding task presents for computational models both for language and vision.

CogSci 2021

A Computational Model of Comprehension in Manga Style Visual Narratives

### Looking for Cogsci 2023? Please [click here](https://underline.io/events/405/reception) to switch to 2023 edition.


**Dear Colleagues,**

Welcome to the 43rd Annual Conference of the Cognitive Science Society! 

This year’s conference brings together the latest research from all Cognitive Science and also highlights this year’s theme **Comparative Cognition: Animal Minds.** 

The program features four plenary speakers: Friederike Range (University of Veterinary Medicine, Vienna), Onur Güntürkün (Ruhr University Bochum), Julia Fischer (University of Göttingen), and Erich Jarvis (The Rockefeller University). The keynotes and invited symposia showcase the broad spectrum of ideas that encompass the conference theme of comparative cognition. 

Further, the program includes the Rumelhart Prize keynote presentation by Susan Goldin-Meadow (University of Chicago), and symposia to celebrate the winner of the Elman Prize: Wei Ji Ma (New York University), the Heineken Award winner Robert Zatorre (McGill University), and the Robert J. Glushko Dissertation Prizes session.

The program committee for CogSci 2021 received 1,025 submissions, including 710 full papers, 275 member abstracts, 8 publication-based talks, as well as 15 proposals for symposia, and 17 for workshops and tutorials. After a rigorous review process, the committee selected 152 papers for oral presentation and inclusion in the conference proceedings (14.8%), 332 papers for poster presentation and inclusion in the proceedings (32.4%), and 175 papers for poster presentation with inclusion of abstracts in the proceedings (17.1%) We also selected 275 submitted member abstracts. In addition, we accepted 6 publication-based talks, 12 symposia, and 9 workshops and tutorials to make for an exciting and inclusive program.

We hope that you enjoy the conference this year!<br><br>

**Your Program Chairs,**
![](https://assets.underline.io/uploads/markdown_image/1/image/99984908caba5e6453e5fc5ba34f3507.jpg)

**Learn more about the Vienna Cognitive Science Hub at the University of Vienna, hosts of CogSci 2021**

<iframe width="560" height="315" src="https://www.youtube.com/embed/464h1m3iU68" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

To review the proceedings for the conference or review the searchable program, please visit the [website](https://cognitivesciencesociety.org/program/).


The Cognitive Science Society brings together researchers from around the world who hold a common goal: understanding the nature of the human mind. The mission of the Society is to promote Cognitive Science as a discipline, and to foster scientific interchange among researchers in various areas of study, including Artificial Intelligence, Linguistics, Anthropology, Psychology, Neuroscience, Philosophy, and Education. The Society is a non-profit professional organization and its activities include sponsoring an annual conference and publishing the journals Cognitive Science and TopiCS.

poster

Due to information processing constraints and cognitive limitations, humans must form limited representations of complex decision making tasks. However, the mechanisms by which humans generate representations of task-relevant stimulus remain unclear. We develop a model that seeks to account for the formation of these representations using a β-variational autoencoder (β-VAE) trained with a novel utility based learning objective. The proposed model forms latent representations of decision making tasks that are constrained in their information complexity. We show through simulation that this approach can account for important phenomena in human economic decision making tasks. This model provides a method of forming task-relevant representations that can be used to make decisions in a human-like manner.

Modeling Capacity-Limited Decision Making Using a Variational Autoencoder

The study of emergent languages in deep multi-agent simulations
has become an important research field. While targeting
different objectives, most studies focus on analyzing properties
of the emergent language—often in relation to the agents’
inputs—ignoring the influence of the agents’ perceptual processes.
In this work, we use communication games to investigate
how differences in perception affect emergent language.
Using a conventional setup, we train two deep reinforcement
learning agents, a sender and a receiver, on a reference game.
However, we systematically manipulate the agents’ perception
by enforcing similar representations for objects with specific
shared features. We find that perceptual biases of both sender
and receiver influence which object features the agents’ messages
are grounded in. When uniformly enforcing the similarity
of all features that are relevant for the reference game,
agents perform better and the emergent protocol better captures
conceptual input properties.

Why and how to study the impact of perception on language emergence in artificial agents

We model the sense-making process of diagrams as conceptual blends of the diagrams’ geometric configurations with apt image schemas.We specify image schemas and geometric configurations with typed FOL theories. In addition, for the latter, we utilise some Qualitative Spatial Reasoning formalisms. Using an algebraic specification language, we can compute the conceptual blends of image schemas and geometry as category-theoretic colimits.We show through several examples how this model captures the sort of direct inferences we confer to diagrammatic representations due to our embodied cognition. We argue that this approach to sense-making might be of value for the design and application of diagrammatic and graphical visualisations, as well as for AI in general.

Modelling the Sense-Making of Diagrams Using Image Schemas

Most theoretical accounts of non-instrumental information seeking suggest that the magnitude of rewards has a direct influence on the attractiveness of the information. Specifically, the magnitude of rewards is assumed to be proportional to the strength of information seeking (or avoidant) behaviour. In a series of experiments using numerical and pictorial stimuli, we explore the extent to which observed information seeking behaviour tracks these predictions. Our findings indicate a robust independence of information seeking from outcome magnitude and valence with preferences for information largely remaining constant across different reward valence and magnitudes. We discuss these results in the context of current computational models with suggestions for future theoretical and empirical work.

Invariance of Information Seeking Across Reward Magnitudes

One of the consequences of the pandemic is that throughout 2020 virtual interactions largely replaced face-to-face interactions. Though there are few studies of how social media impacts body image perception across genders, research suggests that socializing through a virtual self-body image might have distinct implications for men and women. On an online study, we examined whether type of social pressure and body-ideal exert distinct pressures on members of the X, Y, and Z generations. Results showed media pressure affected body image satisfaction significantly more than other kinds of social pressure across genders and generations, with young males reporting a higher impact compared to older males. Males experienced more pressure to be muscular and women to be thin, especially for the younger generation.Future research should focus on social media as a potential intervention tool for the detection and prevention of body image disorders in both young female and male adults.

Body Image During Quarantine; Generational Effects of Social Media Pressure on Body Appearance Perception

It is an ancient insight that human actions are driven by desires. Yet it misses one mental representation, intention, with which agents regulate conflicting desires by committing to an admissible plan. Here we demonstrate four behavioral signatures of intention only observed in humans: disruption resistance as sticking with a plan despite setbacks; exclusiveness as avoiding paths with temptations of re-planning; deliberation as the gradual emergence of a commitment plan; temporal leap as forming future plans before finishing the current one. Humans were compared against an optimal model formulated as Markov Decision Process (MDP), who acts only to maximize expected future rewards. Conflicting desires are defined as a reward function returning positive rewards for multiple states. It showed none of the behavioral signatures of intention. These results reveal that humans regulate conflicting desires with intentions, which directly drive actions.

Intention beyond Desire: Humans Spontaneously Commit to Future Actions

Deep Neural Network representations correlate very well with neural responses measured in primates' brains and with psychological representations of human similarity judgement tasks, making them possible models for human behavior-related tasks. This study investigates whether DNNs can learn an implicit association (between colors and emotions) for images. An experiment was conducted in which subjects were asked to select a color for a given emotion-inducing image. These human responses (decision probabilities) were modeled on neural networks using representations extracted from pre-trained DNNs for the images and colors (a square of the color). The model presented showed a fuzzy linear relationship with the decision probabilities. Finally, this model was presented as a model for emotion classification tasks, specifically with very few training examples, showing an improvement in accuracy from a standard classification model. This analysis can be of relevance to psychologists studying these associations and AI researchers modelling emotional intelligence in machines.

Emotion-Color Association in Biologically Inspired Deep Neural Networks

A critical aspect of the way humans recognize and understand meaning in sequential data is the ability to identify abstract structural repetitions. We present a novel approach to discovering structural repetitions within sequences that uses a multiple Smith-Waterman self-alignment. We illustrate our approach in the context of finding different forms of structural repetition in music composition. Feature-specific alignment scoring functions enable structure finding in primitive features such as rhythm, melody, and lyrics. These can be compounded to create scoring functions that find higher-level structure including verse-chorus structure. We demonstrate our approach by finding harmonic, pitch, rhythmic, and lyrical structure in symbolic music and compounding these viewpoints to identify the abstract structure of verse-chorus segmentation.

Inferring Structural Constraints in Musical Sequences via Multiple Self-Alignment

It is a long standing challenge to devise a formal model of ACT-R as a basis for formal reasoning on ACT-R. The ACT-R architecture is a composition of components (such as modules) with predefined interfaces between components and predefined interactions on the interfaces. Reasoning over the correctness of a formal model of ACT-R benefits from the separation of abstraction levels i.e. reasoning on the level of interfaces and interactions between components in isolation from the concrete behaviour of each component. 
We propose a formal semantics of ACT-R that preserves the structural properties of architectural components, i.e., the interfaces of modules to the remaining architecture as well as communication between modules within the architecture. We demonstrate how our new formal semantics of ACT-R serves to prove the correctness of the timed automaton based operational semantics for ACT-R (TA-ACT-R) on the level of architectural components.

A Formal Operational Model of ACT-R: Structure and Behaviour

Humans are highly efficient learners, with the ability to grasp the meaning of a new concept from just a few examples. Unlike popular computer vision systems, humans can flexibly leverage the compositional structure of the visual world, understanding new concepts as combinations of existing concepts. In the current paper, we study how people learn different types of visual compositions, using abstract visual forms with rich relational structure. We find that people can make meaningful compositional generalizations from just a few examples in a variety of scenarios, and we develop a Bayesian program induction model that provides a close fit to the behavioral data. Unlike past work examining special cases of compositionality, our work shows how a single computational approach can account for many distinct types of compositional generalization.

A Computational Model of Comprehension in Manga Style Visual Narratives

Downloads

Next from CogSci 2021

Modeling Capacity-Limited Decision Making Using a Variational Autoencoder

Similar lecture

Behavioral interference or facilitation does not distinguish between competitive and noncompetitive accounts of lexical selection in word production