United States

Multimodal machine translation leverages multiple data modalities to enhance translation quality, particularly for low-resourced languages. This paper uses a multimodal model that integrates visual information with textual data to improve translation accuracy from English to Hindi, Malayalam, Bengali, and Hausa. This approach employs a gated fusion mechanism to effectively combine the outputs of textual and visual encoders, enabling more nuanced translations that consider both language and contextual visual cues. The model&#39;s performance was evaluated against the text-only machine translation model based on BLEU, ChrF2 and TER. Experimental results demonstrate that the multimodal approach consistently outperforms the text-only baseline, highlighting the potential of integrating visual information in low-resourced language translation tasks.

EMNLP 2024

English-to-Low-Resource Translation: A Multimodal Approach for Hindi, Malayalam, Bengali, and Hausa

multimodal

low-resource

machine translation

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

Multimodal machine translation leverages multiple data modalities to enhance translation quality, particularly for low-resourced languages. This paper uses a multimodal model that integrates visual information with textual data to improve translation accuracy from English to Hindi, Malayalam, Bengali, and Hausa. This approach employs a gated fusion mechanism to effectively combine the outputs of textual and visual encoders, enabling more nuanced translations that consider both language and contextual visual cues. The model's performance was evaluated against the text-only machine translation model based on BLEU, ChrF2 and TER. Experimental results demonstrate that the multimodal approach consistently outperforms the text-only baseline, highlighting the potential of integrating visual information in low-resourced language translation tasks.

This paper covers the system description of the team “ODIAGEN’s” submission to the 11th Workshop on Asian Translation (WAT 2024). We participated in the English-to-LowRes Multimodal Translation Task, in two of the tasks, i.e. Textonly Translation and Multi-modal Translation. For Text-only Translation, we trained the Mistral-7B model for English to Multi-lingual (Hindi, Bengali, Malayalam, Hausa). For Multi-modal Translation (using both image and text), we trained the PaliGemma-3B model for English to Hindi translation.

OdiaGenAI’s Participation in WMT2024 English-to-Low Resource Multimodal Translation Task

We present our submission to the WMT24 shared task for Low-Resource Languages of Spain in the constrained category. With limited deep learning-based filtration tools for these languages, we developed a purely statistical, two-stage data filtration pipeline. First, we preprocess by removing spaces, punctuation, and duplicates from the Spanish source sentences, followed by filtering based on inconsistent language predictions and sentence-length anomalies. In the second stage, for larger corpora, we employ a Jensen-Shannon divergence method to curate the final training set. Our approach enabled us to train our model in under 3 hours, with GPU consumption below 1 kWh, making it both cost-effective and eco-friendly. Full details, including code and models, are available on our GitHub.

Back to the Stats: Rescuing Low Resource Neural Machine Translation with Statistical Methods

Our work on the WMT24 Shared Task focuses on improving machine translation for low-resource languages of Spain, specifically Aragonese, Asturian, and Aranese. We utilized a distilled NLLB-200 model with added language-specific tokens and a carefully curated data pipeline. Despite limited resources, our TRIBBLE model achieved results close to the Apertium baseline in Asturian, though challenges remain for Aragonese and Aranese.

TRIBBLE - TRanslating IBerian languages Based on Limited E-resources

This paper describes the CloudSheep translation system for WMT24 Discourse-Level Literary Translation shared task. We participated in the Chinese-English direction on the unconstrained track. Our approach to the task used a pipeline of different tools in order to maximize the translation accuracy and flow of the text by combining the strengths of each tool. In particular, our focus was to translate names consistently and idioms correctly. To achieve consistent names throughout a text, a custom name dictionary was generated for each text, containing person and place names, along with their translations. A common honorific dictionary was applied for consistency with titles, especially in historical or cultivation novels. The names were found and translated with GPT 3.5-turbo. To achieve accurate and concise translations of idioms, which are often translated literally and verbosely, we integrated the CC-CEDICT library to provide official definitions. Then, we used GPT-4 to pick the best dictionary definition that fit the context and rephrase it to fit grammatically within a sentence. For the translation of non-name and non-idiom terms, we used Google Translate. We compared our approach's performance with Google Translate as a baseline using BLEU, chrF, and COMET, as well as A/B testing.

CloudSheep System for WMT24 Discourse-Level Literary Translation

This poster describes our submission system, NovelTrans, from NLP²CT and DeepTranx for the WMT24 Discourse-Level Literary Translation Task in Chinese-English, Chinese-German, and Chinese-Russian language pairs under unconstrained conditions. For our primary system, three translations are done by GPT4o using three different settings of additional information and a terminology table generated by online models. The final result is composed of sentences that have the highest xCOMET score compared with the corresponding sentences in other results. Our system achieved an xCOMET score of 79.14 which is higher than performing a direct chapter-level translation on our dataset.

NovelTrans: System for WMT24 Discourse-Level Literary Translation

The rapid growth of deep learning has spurred significant advancements across industries, particularly in machine translation through large languagemodels (LLMs). However, translating literary still presents challenges, including cross-cultural nuances, complex language structures, metaphoricalexpressions, and cultural differences. To address these issues, this study utilizes the Llama and Phi models using both LoRA and full-parametertechniques, along-side a prompt-based translation system.

LinChance×NTU for Unconstrained WMT2024 Literary Translation

Chat text translation in multinational customer service faces unique challenges such as colloquial expressions, abbreviations, and context-dependent meanings. This study proposes a context-aware sliding window mechanism to enhance translation quality. Through ablation studies, we compare multilingual and bilingual fine-tuning approaches and evaluate the impact of window sizes on translation performance. Our findings demonstrate how contextual information, fine-tuning strategies, and window size influence document-level translation quality, ensuring consistent context handling and robust cross-linguistic performance.

Optimising LLM-Driven Machine Translation with Context-Aware Sliding Windows

Translating conversational text, particularly in customer support contexts, presents unique challenges due to its informal and unstructured nature. We propose a context-aware LLM translation system that leverages conversation summarization and dialogue history to enhance translation quality for the English-Korean language pair. Our approach incorporates the two most recent dialogues as raw data and a summary of earlier conversations to manage context length effectively. We demonstrate that this method significantly improves translation accuracy, maintaining coherence and consistency across conversations. This system offers a practical solution for customer support translation tasks, addressing the complexities of conversational text.

Context-Aware LLM Translation System Using Conversation Summarization and Dialogue History

For this shared task, we have used several ma- chine translation engines to produce transla- tions (en ⇔ fr) by fine-tuning a dialog-oriented NMT engine and having NMT baseline trans- lations post-edited with prompt engineering. Our objectives are to test the effectiveness of a fine-tuning strategy with help of a robust NMT model, to draw out a from-translation-to-post- editing pipeline, and to evaluate the strong and weak points of NMT systems.

Enhancing Translation Quality: A Comparative Study of Fine-Tuning and Prompt Engineering in Dialog-Oriented Machine Translation Systems. Insights from the MULTITAN-GML Team

Our approach focused on fine-tuning Large Language Models (LLMs), using in- domain and synthetic data, employing different data retrieval strategies. We built our MT systems using multilingual LLMs such as NLLB-200-600M, Llama-3-8b and mBART-50. We present our findings and report the quality of the outputs.

Downloads

Next from EMNLP 2024

OdiaGenAI’s Participation in WMT2024 English-to-Low Resource Multimodal Translation Task

Similar lecture

OdiaGenAI’s Participation in WMT2024 English-to-Low Resource Multimodal Translation Task

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2024

OdiaGenAI’s Participation in WMT2024 English-to-Low Resource Multimodal Translation Task

Similar lecture

OdiaGenAI’s Participation in WMT2024 English-to-Low Resource Multimodal Translation Task

Downloads