United States

This study examines the reliability and validity of using two types of crowd-sourced judgments to collect lexical diversity scores. Scaled and pairwise comparison approaches were used to collect data from non-expert Amazon Mechanical Turk workers. The reliability of the lexical diversity ratings for the crowd-sourced raters was assessed along with those from trained raters using a variety of reliability statistics. The validity of the ratings was examined by 1) comparing crowd-sourced and trained ratings, 2) comparing crowd-sourced and trained ratings to ratings of language proficiency, and 3) by using an objective measure of lexical diversity to predict the crowd-sourced and trained ratings. The results indicate that scaled crowd-sourced ratings showed strong reliability in terms of text and rater strata and showed fewer misfitted texts than the trained raters. The scaled crowd-sourced ratings were also strongly predicted by lexical diversity features derived from the texts themselves.

CogSci 2023

Crowd-Sourcing Human Ratings of Linguistic Production

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP 2020

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


poster

The 45th Annual Meeting of the Cognitive Science Society will take place in Sydney, Australia as a modified hybrid conference. Presentations from the main-session room will be live-streamed every day for all events. This will include all invited talks and award sessions. Recordings of all CogSci 2023 sessions will be made available for on-demand viewing after the conference.<br>
The in-person component will be held at the International Convention Centre Sydney at Darling Harbour. The Conference will take place Wednesday July 26th – Saturday July 29th, 2023 (Wednesday, July 26th is the Workshops Day).

Join us!

<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2023/07/COgSci-PAAG-V2.pdf)
<br>[**The Detailed Program can be found here**](https://online.fliphtml5.com/ebtyf/mswz/)
<br><br>

#### **CogSci 2023 Program Book**
<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/tzgk/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>
<br>

The 45th Annual Meeting of the Cognitive Science Society presents the latest research across cognitive science and highlights the theme of Cognition in Context.

Words that regularly fill the same sentential slots are said to be paradigmatically related. Paradigmatic relations may be retained through a direct association or a latent representation at encoding, or by reinstating context during retrieval. We paired proper names by embedding them into two instances of the same sentence frame, each in a separate list, yielding blocks of two study-cloze sessions. The pairing between proper names was fixed across twelve blocks. In the static condition, the same sentence frames were used across blocks, while in the dynamic condition sentence frames changed for each block. Interference should accrue in both conditions if paradigmatic relations are based on a direct association or overlap in a latent representation, however, if paradigmatic relations are mediated by retrieved context then changing the sentence frame should release interference. Our results are consistent with a context-mediation account of paradigmatic relations.

Paradigmatic formation through context-mediation

What makes some sentences more memorable than others? In this work, we build on past work in image and word memorability and conduct a large-scale sentence memorability experiment with responses from 500 participants for 2,500 target sentences. We find that sentences vary in their intrinsic memorability. We predict the memorability of sentences from a variety of word- and sentence-level features, including representations from artificial large language models and the information-theoretic measure of surprisal. We find that the best predictor of sentence memorability is the presence of individually memorable words, but that in the absence of such words, sentence-level features predict sentence memorability. This suggests that otherwise forgettable words can together create memorable compositional meanings that remain in memory.

Context-sensitive features predict sentence memorability in the absence of memorable words

The relevance of Broca's area to language grammar processing is well established. Its relevance to implicitly learned non-linguistic sequential rules has also been demonstrated. Previous work by our lab has shown this is true even if subjects are unaware they are being trained on, and tested for, sensitivity to these rules. We extend this work to show that the degree of differential activation in Broca's area for implicitly learned grammatical vs. ungrammatical sequences is correlated across subjects with their behavioral evidence of their degree of learning of these rules as assessed by reaction time differences. Broca's is among the three regions showing the highest association with degree of learning. This finding further underscores the relevance of Broca's for non-linguistic rule learning, and suggests that the original function of this area in our pre-human ancestors may have been the implicit learning of any kind of sequential patterns in the environment.

Degree of learning of an artificial grammar correlates with differential fMRI activation of Broca's area

Reference to events using pronouns like 'it' or demonstratives like 'that' has been difficult to study, because unlike in reference to people or objects, the ground truth of interpretation is harder to establish. In this paper, we introduce a new task to understand the roles of three parameters in event reference resolution: The referential expressions themselves, sentential aspect, and discourse structure. We find that different referring expressions reliably refer to specific parts of a discourse, and confirm previous findings that sentential aspect influences referential accessibility; that the structure of a discourse itself has a major effect on reference resolution, and most notably that, contrary to predictions in the literature, some events that are not on the right-frontier of a discourse are nevertheless available for reference. Our findings contribute to a growing literature on anaphor resolution that finds that the parameter structure that determines reference resolution is multifaceted, but predictable.

Discourse structure affects reference resolution to events in English: Evidence from a new paradigm

Repetition of linguistic structure plays a role in both language comprehension and production. Previously encountered structures are processed faster, and speakers tend to reuse them in new utterances—a phenomenon known as structural priming. According to the prevailing interpretation of structural priming, linguistic input activates an underlying mental representation based on constituents, a syntactic unit derived from rule-based grammars (e.g., [[he]NP [hears [a sound]NP]VP]S). Here we ask whether structural priming can occur for non-constituent parts-of-speech fragments, such as pronoun verb determiner (e.g., he hears a). Across two preregistered phrasal decision experiments, we show that structural priming can occur at the level of three-word parts-of-speech sequences and in the absence of constituents. Using corpus analysis, we further show that structural priming of non-constituents also occurs in real-life dialogue. These results imply that constituent structure is not a necessary prerequisite for structural priming and provide a challenge to contemporary approaches to grammar.

Does he hears a prime I prefer the? Testing a novel chunk-based perspective on structural priming

The lesser known G(rammatical)-Maze task (Forster, Guerrera, & Elliot, 2009) is arguably a better choice than self-paced reading (Mitchell, 2004) for detecting difficulty from word to word in online sentence processing over crowdsourcing platforms. In G-Maze, a participant must choose between each successive word in a sentence and a distractor word that does not make sense based on the preceding context. If a participant chooses the distractor as opposed to the actual word, then the trial ends and they may not complete the sentence. Thus, G-Maze automatically filters out data from inattentive participants, and more effectively localizes differences in processing difficulty. Still, the effort required to pick contextually inappropriate distractors for hundreds of words might cause an experimenter to hesitate before picking this method. To save experimenters this time and effort, Boyce, Futrell, and Levy (2020) developed A(uto)-Maze, a tool that automatically generates distractors using a computational language model. We now introduce the next generation of A-Maze: T(ransformer)-Maze. Transformer models are the current state of the art in natural language processing, and thousands, pretrained in a variety of languages, are freely available on the internet, specifically through Huggingface’s Transformers package (Wolf et al., 2020). In our validation experiment, T-Maze proves itself to be as effective as G-Maze with handmade materials, run in a lab. This tool thus allows psycholinguists to easily gather high-quality online sentence processing data in many different languages.

Transformer-Maze: An Accessible Incremental Processing Measurement Tool

Logical connective words in natural languages often have multiple interpretations. A disjunction can be exclusive (XOR) or inclusive (IOR), and a negative conjunction can be interpreted as alternative denial (NAND) or joint denial (NOR). Previous research has suggested that there is substantial crosslinguistic variation in the interpretation of such logical constructions. That languages fall into two groups with different default interpretations. The first group interprets a negative conjunction as alternative denial (NAND) and a negative disjunction as joint denial (NOR). The second group has the opposite default interpretations. Using an online card selection task, we tested speakers of English, Hungarian, Spanish, and Mandarin on a wide variety of logical constructions. We found that the speakers of these languages did not show considerable variation in their interpretations of the logical constructions tested. The results suggest that the interpretation of logical constructions may be more consistent crosslinguistically than previously hypothesized.

Crosslinguistic Consistency in the Interpretation of Logical Connectives: The case of English, Hungarian, Spanish, and Mandarin Chinese

Visual attention can influence how language is processed. For instance, previous research showed that visual cueing of a referent affects the choice of a particular grammatical structure. The present study extended this research to language comprehension and investigated whether visually cueing either the subject or the object of an event affects the interpretation of a textual event description in Bulgarian with either a subject-initial or object-initial word order. The presence of the visual cue decreased self-paced reading reaction times and altered the accuracy in response to a comprehension question. Differences were found depending on the gender of the referent, highlighting the interaction of the visual cue and other linguistic information such as case marking. This study demonstrates that visually induced "context" can affect the linguistic salience of a referent in discourse and illustrates the interaction of domain-general cognitive mechanisms in language processing.

Visual cueing affects the processing of grammatical structure: A self-paced reading study on non-canonical word order in Bulgarian

Is there a cognitive bias against absolute synonymy? The current work explored this question via a miniature mini-artificial language experiment featuring iterated learning, which can amplify weak cognitive biases, as languages are sifted through multiple adult learners. Participants were taught two novel synonymous verbs in positive or negative sentences. Afterwards, they had to generalize the words to new positive or
negative sentences. These sentences then served as the input sentences for the next participant in the diffusion chain and so on. Despite inconsistent input with regard to positive or negative meaning, participants differentiated the verbs. More than half of participants strongly differentiated them, specializing at least one term. However, transmission did not increase differentiation overall, suggesting that a bias against synonymy may encourage a minimally distinctive difference, not necessarily a systematic one, between synonyms.

A Weak Bias Against Strong Synonymy?

Consider the following two (hypothetical) generic causal claims: “Attending an all-girls school improves girls’ math scores” and “attending an affluent all-girls school improves girls’ math scores.” These claims not only differ in what they suggest about how test scores are distributed across different types of schools (i.e., “the data”), but also have the potential to communicate something about the speakers’ values: namely, the prominence they accord to affluence in representing and making decisions about the social world. Here, we examine the relationship between the level of granularity with which a cause is described in a generic causal claim (e.g., all-girls school vs. affluent all-girls school) and the value of the information contained in the causal model that generates that claim. We argue that listeners who know any two of the following can make reliable inferences about the third: 1) the level of granularity at which a speaker makes a generic causal claim, 2) the speaker’s decision-theoretic values, and 3) the data available to the speaker. We present results of three experiments in the domain of social categories (N=853) that provide evidence in keeping with these predictions.

Tell Me Your (Cognitive) Budget, and I’ll Tell You What You Value: Evidential Relationships Between Values, Data, and Generic Causal Claims about the Social World

Premium content

Downloads

Next from CogSci 2023

Paradigmatic formation through context-mediation

Similar lecture

The Challenges of Crowd Workers in Rural and Urban America