Dominican Republic

While powerful pre-trained language models have improved the fluency of text generation models, it remains difficult to ensure that the generated texts are semantically faithful to the input. In this paper, we introduce a novel automatic evaluation metric, Entity-Based Semantic Adequacy, which can be used to assess to what extent generation models that verbalise RDF (Resource Description Framework) graphs produce text that contains mentions of the entities occurring in the RDF input. This is important as RDF subject and object entities make up 2/3 of the input. We use our metric to compare 25 models from the WebNLG Shared Tasks and we examine correlation with results from human evaluations of semantic adequacy.
We show that while our metric correlates with human evaluation scores,
this correlation varies with the specifics of the human evaluation setup. This suggests that in order to measure the entity-based adequacy of generated texts, an automatic metric such as the one proposed here might be more reliable, as less subjective and more focused on correct verbalisation of the input, than human evaluation measures.

EMNLP 2021

Entity-Based Semantic Adequacy for Data-to-Text Generation

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

workshop paper

SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the
patterns of the contributions behind SemEval. By understanding the distribution
of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Most of the time, when dealing with a particular Natural Language Processing task, systems are compared on the basis of global statistics such as recall, precision, F1-score, etc. While such scores provide a general idea of the behavior of these systems, they ignore a key piece of information that can be useful for assessing progress and discerning remaining challenges: the relative difficulty of test instances. To address this shortcoming, we introduce the notion of differential evaluation which effectively defines a pragmatic partition of instances into gradually more difficult bins by leveraging the predictions made by a set of systems. Comparing systems along these difficulty bins enables us to produce a finer-grained analysis of their relative merits, which we illustrate on two use-cases: a comparison of systems participating in a multi-label text classification task (CLEF eHealth 2018 ICD-10 coding), and a comparison of neural models trained for biomedical entity detection (BioCreative V chemical-disease relations dataset).

Differential Evaluation: Assessing Natural Language Processing SystemPerformance Based Upon Data Resistance to Processing

In this paper, we introduce a new embedding-based metric relying on trainable ranking models to evaluate the semantic accuracy of neural data-to-text generators.
This metric is especially well suited to semantically and factually assess the performance of a text generator when tables can be associated with multiple references and table values contain textual utterances.
We first present how one can implement and further specialize the metric by training the underlying ranking models on a legal Data-to-Text dataset.
We show how it may provide a more robust evaluation than other evaluation schemes in challenging settings using a dataset comprising paraphrases between the table values and their respective references.
Finally, we evaluate its generalization capabilities on a well-known dataset, WebNLG, by comparing it with human evaluation and a metric recently introduced based on natural language inference.
We then illustrate how it naturally characterizes, both quantitatively and qualitatively, omissions and hallucinations.

Trainable Ranking Models to Evaluate the Semantic Accuracy of Data-to-Text Neural Generator

We present the joint contribution of Instituto Superior Técnico (IST) and Unbabel to the Explainable Quality Estimation (QE) shared task, where systems were submitted to two tracks: constrained (without word-level supervision) and unconstrained (with word-level supervision).

IST-Unbabel 2021 Submission for the Explainable Quality Estimation Shared Task

This presentation describes the UMD submission to the Explainable Quality Estimation Shared Task at the Eval4NLP 2021 Workshop on ``Evaluation & Comparison of NLP Systems''. We participated in the word-level and sentence-level MT Quality Estimation (QE) constrained tasks for all language pairs: Estonian-English, Romanian-English, German-Chinese, and Russian-German. Our approach combines the predictions of a word-level explainer model on top of a sentence-level QE model and a sequence labeler trained on synthetic data. These models are based on pre-trained multilingual language models and do not require any word-level annotations for training, making them well suited to zero-shot settings. Our best performing system improves over the best baseline across all metrics and language pairs, with an average gain of 0.1 in AUC, Average Precision, and Recall at Top-K score.

The UMD Submission to the Explainable MT Quality Estimation Shared Task: Combining Explanation Models with Sequence Labeling

Current research on quality estimation of machine translation focuses on the sentence-level quality of the translations. By using explainability methods, we can use these quality estimations for word-level error identification. In this work, we compare different explainability techniques and investigate gradient-based and perturbation-based methods by measuring their performance and required computational efforts. Throughout our experiments, we observed that using absolute word scores boosts the performance of gradient-based explainers significantly. Further, we combine explainability methods to ensembles to exploit the strengths of individual explainers to get better explanations. We propose the usage of absolute gradient-based methods. These work comparably well to popular perturbation-based ones while being more time-efficient.

Explaining Errors in Machine Translation with Absolute Gradient Ensembles

Quality Estimation (QE) for Machine Translation has been shown to reach 
relatively high accuracy in predicting sentence-level scores, relying on 
pretrained contextual embeddings and human-produced quality scores. 
However, the lack of explanations along with decisions made by end-to-
end neural models makes the results difficult to interpret. Furthermore, 
word-level annotated datasets are rare due to the prohibitive effort 
required to perform this task, while they could provide interpretable 
signals in addition to sentence-level QE outputs. In this paper, we 
propose a novel QE architecture which tackles both the word-level data 
scarcity and the interpretability limitations of recent approaches. 
Sentence-level and word-level components are jointly pretrained through 
an attention mechanism based on synthetic data and a set of MT metrics 
embedded in a common space. Our approach is evaluated on the Eval4NLP 
2021 shared task and our submissions reach the first position in all 
language pairs. The extraction of metric-to-input attention weights show 
that different metrics focus on different parts of the source and target 
text, providing strong rationales in the decision-making process of the 
QE model.

Error Identification for Machine Translation with Metric Embedding and Attention

Anticipating the rise in Artificial Intelligence’s ability to produce original works of literature, this study suggests that literariness, or that which constitutes a text as literary, is understudied in relation to text generation. From a computational perspective, literature is particularly challenging because it typically employs figurative and ambiguous language. Literary expertise would be beneficial to understanding how meaning and emotion are conveyed in this art form but is often overlooked. We propose placing experts from two dissimilar disciplines – machine learning and literary studies – in conversation to improve the quality of AI writing. Concentrating on evaluation as a vital stage in the text generation process, the study demonstrates that benefit could be derived from literary theoretical perspectives. This knowledge would improve algorithm design and enable a deeper understanding of how AI learns and generates.

This presentation is based on the following article: 
Imke van Heerden and Anil Bas, AI as Author – Bridging the Gap Between Machine Learning and Literary Theory, Journal of Artificial Intelligence Research, 71, pp. 175–189, 2021.

AI as Author – Bridging the Gap Between Machine Learning and Literary Theory

Estimating the expected output quality of generation systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system-level quality. Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposition, we adjust this error to a noise-free, infinite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in variance: when the number of human judgments used is small, and when the quality difference between compared systems is small.

The statistical advantage of automatic NLG metrics at the system level

Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40 percentage points on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.

Downloads

Next from EMNLP 2021

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Similar lecture

Entity-Based Semantic Adequacy for Data-to-Text Generation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2021

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Similar lecture

Entity-Based Semantic Adequacy for Data-to-Text Generation

Downloads