United Arab Emirates

Most commercial conversational AI products in domains spanning e-commerce, health care, finance, and education involve a hierarchy of NLP models that perform a variety of tasks such as classification, entity recognition, question-answering, sentiment detection, semantic text similarity, and so on. Despite our understanding of each of the constituent models, we do not have a clear view as to how these models affect the overall platform metrics. To bridge this gap, we define a metric known as answerability, which penalizes not only irrelevant or incorrect chatbot responses but also unhelpful responses that do not serve the chatbot&#39;s purpose despite being correct or relevant. Additionally, we describe a formula-based mathematical framework to relate individual model metrics to the answerability metric. We also describe a modeling approach for predicting a chatbot&#39;s answerability to a user question and its corresponding chatbot response.

EMNLP 2022

Answerability: A custom metric for evaluating chatbot performance

dialogue systems

question answering

transformers

Congratulations to all the researchers who have qualified for the AMA Research Challenge virtual poster symposium and semifinals!

Welcome to the **largest national, multi-specialty research event** featuring research from nearly 1,000 medical students, residents, fellows, and international medical graduates across six topic areas - basic science, clinical and translational research, clinical vignettes, health systems science, medical education, and public health and health policy.

#### Access the Event
Use the buttons below or navigation menu to access the event. 

[![](https://assets.underline.io/markdown_image/1/image/d4076f2146939317a29eb9f69c53ad41.png)](https://underline.io/events/506/sessions?searchGroup=all)
[![](https://assets.underline.io/markdown_image/1/image/6bcee0b523a4874c4292ca0415d88dde.png)](https://underline.io/events/506/posters?searchGroup=all)

#### Event Details
The 2025 AMA Research Challenge includes the following symposium halls:
* **Semifinals hall:** Features research from 54 medical students, residents and international medical graduates that received top scores during first round scoring. 
* **Poster symposium hall:** Features research from nearly 1,000 medical students, residents and international medical graduates categorized by topics and subtopics.

**Here’s how to participate:**
* **View posters and presentations** on a variety of topics and specialties
* **Score posters in the semifinals** to help decide the top five finalists (AMA member exclusive opportunity)
* **Connect with presenters** and provide feedback to help advance their research
* **New this year: Watch live poster presentations** by topic area and engage directly with researchers

At the conclusion of this event, the AMA announced the top five finalists who advanced to the AMA Research Challenge final round, presented their research to an elite panel of judges, and competed to win a $10,000 grand prize presented by Laurel Road.

**Addison Shenk, a fourth-year medical student from Edward Via Virginia College of Osteopathic Medicine, was the winner of the 2025 AMA Research Challenge and was awarded the grand prize of $10,000, presented by Laurel Road.**

#### A message from Dr. Bobby Mukkamala, AMA President
<div style="position: relative; width: 100%; padding-bottom: 56.25%; height: 0; overflow: hidden;">
  <iframe 
    src="https://underline.io/embed/136894-ama-video?t=0"
    title="AMA Welcome Video"
    allowfullscreen
    style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: 0;">
  </iframe>
</div>


#### Recognized Institutions
In 2025, 1,400 abstracts were submitted across six topic areas representing 174 medical schools and more than 150 residency and health care institutions. [See if your institution is represented this year!](https://drive.google.com/file/d/1eFrTqCmCeobx9pr5drJK2hllvsIpFNOY/view?usp=drive_link)

#### Grand Prize Sponsor
AMA would like to thank Laurel Road, a brand of KeyBank, for sponsoring a $10,000 grand prize for the winner of the AMA Research Challenge. Laurel Road and KeyBank are not involved in winner selection process. The winner will be announced during the AMA Research Challenge final round. Payment of the grand prize will be directly from Laurel Road/KeyBank. See the [AMA Research Challenge Official Rules for terms and conditions](https://www.ama-assn.org/system/files/research-challenge-official-rules.pdf).

[![](https://assets.underline.io/uploads/markdown_image/1/image/611e331d4a0eba0920f240cfc04b4d77.png)](https://www.laurelroad.com/)

#### Code of Conduct
<html>
AMA has a code of conduct for its meetings – we ask all attendees to adhere to this code to participate in this year’s meeting. Please read and <a href="https://www.ama-assn.org/about/code-conduct" target="_blank">learn more here</a> about your conduct requirements.
<br><br>
	</html>
	

#### Semifinals Judges

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Need help? Contact us at <support@underline.io>.

Please log in to explore this event.

If you have used the AMA Research Challenge platform previously, please click the ‘Log-in’ button below. If you do not remember your password, please click “forgot password” and follow prompts to reset your password. After resetting your password, please refresh your webpage to access the platform. 

Register to explore research in a variety of topics. AMA members get exclusive access to score posters in the semifinals. 

**Please use [this link here](https://www.ama-assn.org/member-benefits/events/ama-research-challenge) to register for free. **

Registration Is Required

AMA Research Challenge 2025 

Join the AMA Research Challenge poster symposium and semifinals to explore top research posters and score posters in the semifinals (an AMA member exclusive opportunity).

Most commercial conversational AI products in domains spanning e-commerce, health care, finance, and education involve a hierarchy of NLP models that perform a variety of tasks such as classification, entity recognition, question-answering, sentiment detection, semantic text similarity, and so on. Despite our understanding of each of the constituent models, we do not have a clear view as to how these models affect the overall platform metrics. To bridge this gap, we define a metric known as answerability, which penalizes not only irrelevant or incorrect chatbot responses but also unhelpful responses that do not serve the chatbot's purpose despite being correct or relevant. Additionally, we describe a formula-based mathematical framework to relate individual model metrics to the answerability metric. We also describe a modeling approach for predicting a chatbot's answerability to a user question and its corresponding chatbot response.

workshop paper

## Welcome to EMNLP 2022!
I am delighted to welcome you to EMNLP 2022!  I believe this conference has been complicated beyond any precedent.  Over the past year, it’s been thrilling to see the organization team approach each new puzzle with creativity and enthusiasm.  We hope that those participating in Abu Dhabi as well as those joining remotely will leave the conference feeling newly inspired by the program and newly connected to our ever-growing community.  Following EMNLP 2021 and major NLP conferences since, EMNLP 2022 is “hybrid,” serving both virtual and in-person participants.

Our key innovations for EMNLP 2022 include:

* EMNLP 2022 is “hybrid” in a second sense, as well:  we allowed both direct and rolling review paper submissions, building on the pilot experiment of EMNLP 2021, which considered a small number of ARR submissions. 
* Familiar from NAACL but new to EMNLP, we’ve added an industry track.
* During the conference, “portals” will link virtual poster sessions to in-person conference participants during poster sessions each day.
* The first ACL-family conference in the United Arab Emirates.

<br>*Message from Noah A. Smith, University of Washington and Allen Institute for AI, Seattle, Washington, USA*<br>
***EMNLP 2022 General Chair***
<br><br><br>
[![](https://assets.underline.io/uploads/markdown_image/1/image/9eec7d4a287ee18c278b08229290aa83.png)](https://drive.google.com/file/d/1OlPv6QBeo62VVTughj2jkiLeyHd1WnUt/view)
<br><br>
[![](https://assets.underline.io/uploads/markdown_image/1/image/a3db7a768409f05192210d98601edb25.png)](https://emnlp2022.rocket.chat/)



To access this site you need to register. Please register [here](https://2022.emnlp.org/registration/).

Register here

Welcome!
EMNLP 2022 will take place in Abu Dhabi from December 7th to December 11th, 2022. And it will be held in hybrid mode, both online and offline.

Source code summaries are a vital tool for the understanding and maintenance of source code as they can be used to explain code in simple terms. However, source code with missing, incorrect, or outdated summaries is a common occurrence in production code. Automatic source code summarisation seeks to solve these issues by generating up-to-date summaries of source code methods. Recent work in automatically generating source code summaries uses neural networks for generating summaries; commonly Sequence-to-Sequence or Transformer models, pretrained on method-summary pairs. The most common method of evaluating the quality of these summaries is comparing the machine-generated summaries against human-written summaries. Summaries can be evaluated using n-gram-based translation metrics such as BLEU, METEOR, or ROUGE-L. However, these metrics alone can be unreliable and new Natural Language Generation metrics based on large pretrained language models provide an alternative. In this paper, we propose a method of improving the evaluation of a model by improving the preprocessing of the data used to train it, as well as proposing evaluating the model with a metric based off a language model, pretrained on a Natural Language (English) alongside traditional metrics. Our evaluation suggests our model has been improved by cleaning and preprocessing the data used in model training. The addition of a pretrained language model metric alongside traditional metrics shows that both produce results which can be used to evaluate neural source code summarisation.

Improved Evaluation of Automatic Source Code Summarisation

Many domains and tasks in natural language generation (NLG) are inherently 'low-resource', where training data, tools and linguistic analyses are scarce. This poses a particular challenge to researchers and system developers in the era of machine-learning-driven NLG. In this position paper, we initially present the challenges researchers & developers often encounter when dealing with low-resource settings in NLG. We then argue that it is unsustainable to collect large aligned datasets or build large language models from scratch for every possible domain due to cost, labour, and time constraints, so researching and developing methods and resources for low-resource settings is vital. We then discuss current approaches to low-resource NLG, followed by proposed solutions and promising avenues for future work in NLG for low-resource settings.

Most NLG is Low-Resource: here's what we can do about it

The Semantic textual similarity (STS) task is commonly used to evaluate the semantic representations that language models (LMs) learn from texts, under the assumption that good-quality representations will yield accurate similarity estimates. When it comes to estimating the similarity of two utterances in a dialogue, however, the conversational context plays a particularly important role. We argue for the need of benchmarks specifically created using conversational data in order to evaluate conversational LMs in the STS task. We introduce GiCCS, a first conversational STS evaluation benchmark for German. We collected the similarity annotations for GiCCS using best-worst scaling and presenting the target items in context, in order to obtain highly-reliable context-dependent similarity scores. We present benchmarking experiments for evaluating LMs on capturing the similarity of utterances. Results suggest that pretraining LMs on conversational data and providing conversational context can be useful for capturing similarity of utterances in dialogues. GiCCS will be publicly available to encourage benchmarking of conversational LMs.

GiCCS: A German in-Context Conversational Similarity Benchmark

Prefix-tuning is a parameter-efficient and powerful technique for adapting a pre-trained language model to a downstream application. However, it uses the same dataset-level tuned set of parameters for all examples in the dataset. We extend the framework with a dynamic method, Control Prefixes, which allows for the effective inclusion of input-dependent information, thereby demonstrating how prefix-tuning can be used for controlled text generation tasks. The method incorporates attribute-level learnable representations into different layers of a pre-trained Transformer, enabling the generated text to be guided in a particular direction. We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG). Using only 0.1–2% additional trainable parameters, we show Control Prefixes can even outperform full fine-tuning methods, and present state-of-the-art results on several data-to-text datasets, including WebNLG. We also examine the common case where input-dependent information is unavailable at test time and show Control Prefixes can excel in this setting also.

Control Prefixes for Parameter-Efficient Text Generation

Evaluating open-domain conversation models has been an open challenge due to the open-ended nature of conversations. In addition to static evaluations, recent work has started to explore a variety of per-turn and per-dialog interactive evaluation mechanisms and provide advice on the best setup. In this work, we adopt the interactive evaluation framework and further apply to multiple models with a focus on per-turn evaluation techniques. Apart from the widely used setting where participants select the best response among different candidates at each turn, one more novel per-turn evaluation setting is adopted, where participants can select all appropriate responses with different fallback strategies to continue the conversation when no response is selected. We evaluate these settings based on sensitivity and consistency using four GPT2-based models that differ in model sizes or fine-tuning data. To better generalize to any model groups with no prior assumptions on their rankings and control evaluation costs for all setups, we also propose a methodology to estimate the required sample size given a minimum performance gap of interest before running most experiments. Our comprehensive human evaluation results shed light on how to conduct credible human evaluations of open domain dialog systems using the interactive setup, and suggest additional future directions.

Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup

While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error analysis is receiving increasing attention, with numerous error annotation schemes being proposed for automatically generated text. However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity. In this paper, our aim is to map out recent work on annotating errors in automatically generated text, with a particular focus on error taxonomies. We describe our systematic paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them. Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.

A Survey of Recent Error Annotation Schemes for Automatically Generated Text

Sharing datasets and benchmarks has been crucial for rapidly improving Natural Language Processing models and systems. Documenting datasets' characteristics (and any modification introduced over time) is equally important to avoid confusion and make comparisons reliable. Here, we describe the case of BigPatent, a dataset for patent summarization that exists in at least two rather different versions under the same name. While previous literature has not clearly distinguished among versions, their differences do not only lay on a surface level but also modify the dataset's core nature and, thus, the complexity of the summarization task. While this paper describes a specific case, we aim to shed light on new challenges that might emerge in resource sharing and advocate for comprehensive documentation of datasets and models.

What's in a (dataset's) name? The case of BigPatent

Similarity metrics for text corpora are becoming critical due to the tremendous growth in the number of generative models. These similarity metrics measure the semantic gap between human and machine-generated text on the corpus level. However, standard methods for evaluating the characteristics of these metrics have yet to be established. We propose a set of automatic measures for evaluating the characteristics of semantic similarity metrics for text corpora. Our measures allow us to sensibly compare and identify the strengths and weaknesses of these metrics. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by comparing it to a collection of classical and state-of-the-art metrics. Our measures revealed that recent metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.

Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora

Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators relative to the many untranslated works published around the world. Machine translation (MT) holds potential to complement the work of human translators by improving both training procedures and their overall efficiency. Literary translation is less constrained than traditional MT settings since translators must balance meaning equivalence, readability, and critical interpretability in the target language. This property, along with the complex discourse-level context present in literary texts, also make it more challenging to computationally model and evaluate. To explore literary MT, we collect a dataset (Par3) of non-English novels in the public domain, each aligned at the paragraph level to multiple human and machine-generated English translations. Using Par3, we discover that state-of-the-art automatic MT metrics cannot distinguish reference human translations from machine translated paragraphs, while expert literary translators prefer the former over the latter at a rate of 84%. The experts note that MT outputs contain not only mistranslations, but also discourse-disrupting errors and stylistic inconsistencies. To address these problems, we train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts.

Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature

While overlap-based machine translation evaluation metrics such as BLEU have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words. The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlations with human quality judgments than BLEU, are opaque in comparison. In this paper, we shed light on the behavior of these learned metrics by creating DEMETR, a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of machine translation evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories. We find that learned metrics perform substantially better than string-based metrics on DEMETR. Additionally, learned metrics differ in their sensitivity to various phenomena (e.g., BERTScore is sensitive to codemix but relatively insensitive to gender manipulation, while COMET is much more sensitive to word repetition than to aspectual changes). We will publicly release DEMETR to spur more informed future development of machine translation evaluation metrics.

Downloads

Next from EMNLP 2022

Improved Evaluation of Automatic Source Code Summarisation

Similar lecture

Optimization of a RAG Model Chatbot for Obstetric Medical Education

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2022

Improved Evaluation of Automatic Source Code Summarisation

Similar lecture

Optimization of a RAG Model Chatbot for Obstetric Medical Education

Downloads