Portugal

The task of translating text to images holds some valid creative potential and has been the subject of study in Computational Creativity. In this paper, we present preliminary work focused on emoji translation. The work-in-progress system is based on techniques of information retrieval. We compare the performance of our system with three deep learning approaches using a text-to-emoji task. The preliminary results suggest some advantages of using a knowledge-base as opposed to a purely data-driven approach. This paper aims to situate the research, underline its relevance and attract valuable feedback for future developments.

ICCC&#39;20

An Approach for Text-to-Emoji Translation

emoji

visual creativity

translation

[![](https://assets.underline.io/uploads/markdown_image/1/image/81a3c0317d24f663b49b996875024d45.png)](https://aclanthology.org/volumes/2022.coling-1/)



### THE CONFERENCE WE KNOW AND WE WANT.
**COLING**, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes but also in industrial research departments including tech-startups. COLING provides opportunities for all these communities to showcase their exciting discovery.

In fall of 2022, COLING will be held in Gyeongju in a hybrid format. All participants can either present at the venue site or join virtually. As more people get vaccinated, we are happy to provide safer environments for our colleagues. We believe that COLING 2022 will be one of the conferences, free from the pandemic. The hybrid format gives presenters and sponsors a valuable opportunity to promote their companies in both an online and in-person venue. For the first time in a long time, customers can interact with their sponsor's products first-hand. The online venue, too, gives sponsors the chance to network with those unable to attend the in-person session.

The hybrid format ultimately lends itself to greater exposure for our sponsors: COLING2022 will let you reach more potential partners and customers than ever before!<br><br><br>

To gain access to this event page you are required to register.<br> Find more information and pay the registration fee on event the organizer’s website **[https://coling2022.org/reg](https://coling2022.org/reg)**

COLING 2022

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for the natural language processing and computational linguistics.

technical paper

Computational creativity is the art, science, philosophy and engineering of computational systems which, by taking on particular responsibilities, exhibit behaviours that unbiased observers would deem to be creative. As a field of research, this area is thriving, with progress in formalising what it means for software to be creative, along with many exciting and valuable applications of creative software in the sciences, the arts, literature, gaming and elsewhere.

The ICCC conference series organized by the Association for Computational Creativity since 2010 is the only scientific conference that focuses on computational creativity alone and also covers all aspects of it.


ICCC'20

The ICCC conference series organized by the Association for Computational Creativity since 2010 is the only scientific conference that focuses on computational creativity alone and also covers all aspects of it.
Computational creativity is the art, science, philosophy and engineering of computational systems which, by taking on particular responsibilities, exhibit behaviors that unbiased observers would deem to be creative. As a field of research, this area is thriving, with progress in formalizing what it means for software to be creative, along with many exciting and valuable applications of creative software in the sciences, the arts, literature, gaming and elsewhere.



A growing number of visual artists use neural networks in their practice. While these networks show promise as an art form, the lack of interpretability limits control to high level decisions based on observations. As an alternative, this research investigates the hand-crafting of network weights coupled with explanatory visualizations as a form of creative control over the internal and lower level processes. Two experimental tools were developed: one for parametrically generating first layer kernels and the second for editing multiple layers. These tools attempt to transform the hand-crafting of features into ``crafting'' in a richer sense by bringing network weights and visual materials into a tight feedback loop. The first author extensively engaged with these tools and these case studies serve to examine the affordances of internal interaction for art-making. The findings suggest that direct manipulation can be used intentionally and can yield insights into network representations, but that hand-crafting networks of greater sophistication would likely require a hybrid approach integrating data-driven methods.

Hand-Crafting Neural Networks for Art-Making

In human-human collaboration, gesture, verbal communications, and emotional responses are among the communication strategies that shape the interactions between collaborators to negotiate the contributions. Emotional feedback allows human collaborators to passively communicate their stance about the experience and convey their perception of the process without distracting the flow of the task. In human-human co-creative collaboration, participants inter-act and contribute to the task based on their perception of the collaboration over time.  In human-AI co-creativity, perceiving the cognitive and emotional state of the user in order to determine the dynamics of collaboration and decide what the agent should contribute to the artifact are two primary challenges of building effective co-creative Artificial Intelligence systems. This paper addresses these challenges with the design of a co-creative agent that can sense the user’s emotional state through facial expression.

ARNY: A Co-Creative System Design Based On Emotional Feedback

We present a general computational approach that enables a machine to generate a dance for any input music. We encode intuitive, flexible heuristics for what a 'good' dance is: the structure of the dance should align with the structure of the music. This flexibility allows the agent to discover creative dances. Human studies show that participants find our dances to be more creative and inspiring compared to meaningful baselines. We also evaluate how perception of creativity changes based on different presentations of the dance. Our code is available at github.com/purvaten/feel-the-music.

Feel The Music: Automatically Generating A Dance For An Input Song

Cross-Domain   creative   systems   can   learn   how   to creatively  map  and/or  transform  semantic  concepts from remote domains to their own. This project sets out to explore how a computational system relies on what it has learned from analyzing images to produce creative work.  Specifically, the system presented here extracts semantic content and themes from an input image and uses this knowledge to creatively generate emotionally-appropriate musical artifacts to accompany the image. An independent experiment was conducted to provide a performance assessment and direct future work.

Emotive Music Composition from Visual Sources of Inspiration

"Flow 1 is the first part of two pieces of music, Flow 1 and Flow 2 
While Flow 1 is more classically inspired, Flow 2 is more experimental.

This piece is influenced by the way in which humans, technology and intelligent systems work together to bring music into being. It works both as personal experiment in compositional style and as a way to bring together tradition and technology to explore the interplay between the composer, the tools they use and they ways in which we can explore this. In all works of this kind there are challenges that the composer has to deal with, using new technologies to invoke feelings is a difficult and thoughtful process. It is important to both be in the ‘flow’ of the music creation process and to understand the ways in which working technology can bend your practices to the ‘workflow’ of the system. 

"

Flow 1

Flow 2 is the second piece in the series of works and is a more experimental piece which is the counterpoint to Flow 1. These pieces are influenced by the way in which humans, technology and intelligent systems work together to bring music into being. It works both as personal experiment in compositional style and as a way to bring together tradition and technology to explore the interplay between the composer, the tools they use and they ways in which we can explore this. In all works of this kind there are challenges that the composer has to deal with, using new technologies to invoke feelings is a difficult and thoughtful process. It is important to both be in the ‘flow’ of the music creation process and to understand the ways in which working technology can bend your practices to the ‘workflow’ of the system.

Flow 2

Generative models for singing voice have been mostly concerned with the task of “singing voice synthesis,” i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. In particular, we outline three such generation schemes, and propose a pipeline to tackle these new tasks. Moreover, we implement such models using generative adversarial networks and evaluate them both objectively and subjectively.

Score and Lyrics-Free Singing Voice Generation

We consider the challenge of designing an artificial agent capable of interacting with humans in collaborative dialogue to produce creative, engaging narratives. Collaborative dialogue is distinct from chit-chat in that it is knowledge building, each utterance provides just enough information to add specificity and reduce ambiguity without limiting the conversation. We use concepts from information theory to define a narrative arc function which models dialogue progression. We demonstrate that this function can be used to modulate a generative conversation model and make it produce more interesting dialogues, compared to baseline outputs. We focus on two modes of modulation: reveal and conceal. Empirically, we show how the narrative arc function can model existing dialogues and shape conversation models towards either mode. We conclude with quantitative evidence suggesting that these modulated models provide interesting and engaging dialogue partners for improvisational theatre performers.

Shaping the Narrative Arc: Information-Theoretic Collaborative Dialogue

The Creative Systems Framework (CSF) formalises creativity as search through a space of concepts.  As a formal account of Margaret Boden’s descriptive hierarchy of creativity, it is at the basis of multiple studies dealing with diverse aspects ofComputational Creativity (CC) systems. However, the CSF at present neither formalises action nor action selection during search, limiting its use in analysing creative processes.  We extend the CSF by explicitly modelling these missing components in the search space traversal function.  We furthermore integrate the distinction between a concept and an artefact, and provide stopping criteria for creative search.  Our extension,  the  Creative  Action  Selection  Framework  (CASF),  is informed by previous studies in CC and draws on concepts from Markov Decision Processes (MDPs). It allows us to de-scribe a creative system as an agent selecting actions based on the value, validity and novelty of concepts and artefacts. TheCASF brings more analytical depth for creative systems that can be modelled as utilising an action selection procedure.

Action Selection in the Creative Systems Framework

"This paper presents a domain independent framework for discussing human–computer co-creativity. It expands on Rhodes’ (1961) four perspectives on creativity
and their later adaptations to socio-cultural views of creativity and computational creativity. The new framework allows the attribution of creativity not only to individual creators but to a collective of creators, recognising the importance of meta-level communication to the creative collaboration, and the variety of creative
contributions that emerge during a co-creative process. It also elaborates on the different communities and contexts surrounding co-creative collaboration and thus facilitates the analysis, evaluation and study of human–computer co-creativity by allowing researchers to describe and situate their work in the field."

An Approach for Text-to-Emoji Translation

Speakers

Downloads

Next from ICCC'20

Hand-Crafting Neural Networks for Art-Making

Similar lecture

FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT