Singapore

Large Language Models (LLMs) are increasingly employed in enterprise question-answering (QA) systems, requiring adaptation to domain-specific knowledge. Among the most prevalent methods for incorporating such knowledge are Retrieval-Augmented Generation (RAG) and fine-tuning (FT). Yet, from a cost–accuracy trade-off perspective, it remains unclear which approach best suits industry scenarios. This study investigates the impact of RAG and FT across two closed, industry-specific datasets, evaluating answer quality and operational costs.We extend the Cost-of-Pass framework proposed by Erol et al. (2025) to jointly assess output quality, generation cost, and user interaction cost. Our findings reveal that while premium models perform best out of the box, open-source models can achieve comparable quality when enhanced with RAG. Overall, RAG emerges as the most effective and cost-efficient adaptation method for closed and open source.

AAAI 2026

Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applications

cost

retrieval augmented generation

question answering

fine-tuning

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community." 

*Message from the General Chair, Dirk Hovy*

EMNLP 2025

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

workshop paper

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

Multi-hop question-answering tasks pose a significant challenge for retrieval-augmented generation (RAG) models, which must effectively retrieve, infer, and reason across multiple documents to provide accurate answers. Despite the support of RAG, large language models (LLMs) often struggle to consistently retrieve the most relevant information from large collections of documents. To address this issue, we propose a novel approach, BrowseNet involving query-specific traversal within a knowledge graph (KG) for multi-hop information retrieval. This method encodes unstructured text data into a KG, where nodes represent document chunks and edges capture lexical relationships between these chunks. By dynamically traversing the knowledge graph based on the type of multi-hop query, the proposed method enhances retrieval performance by leveraging intrinsic network parameters. We evaluate this method against RAG baselines on publicly available multi-hop query datasets. Experimental results demonstrate that BrowseNet establishes a new state-of-the-art retrieval performance in multi-hop retrieval.

Structured Traversal of Knowledge Graphs for Multi-hop Question Answering

Large language models (LLMs) often produce fluent but factually incorrect statements, even when relevant evidence is available, due to misallocation of attention between contextual inputs and parametric knowledge. Ensuring that models actively reason over context and retrieve relevant information is critical for trustworthy and interpretable AI. We introduce COMPASS (Context-Modulated PID Attention Steering System), a lightweight, interpretable framework that dynamically steers attention to retrieved context during generation. Using the Context Reliance Score (CRS), COMPASS identifies which attention heads are underutilizing context, and a PID controller adjusts them in real time to improve evidence grounding and factual consistency. This mechanism enables the model to demonstrate advanced reasoning by actively returning to context and retrieving supporting information when needed, without retraining or multi-pass decoding. Across benchmarks including HotpotQA, XSum, HaluEval, and RAGTruth, COMPASS reduces hallucinations by 2.8–5.8% absolute while revealing how attention heads contribute to context-aligned reasoning. These results show that feedback-driven, interpretable control can enhance reasoning, retrieval, and evidence-based generation in LLMs.

COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation

Retrieval-Augmented Generation (RAG) based on Large Language Models (LLMs) is a powerful solution to understand and query the industry's closed-source documents. However, basic RAG often struggles with complex QA tasks in legal and regulatory domains, particularly when dealing with numerous government documents. The traditional top-$k$ strategy frequently misses golden chunks, leading to incomplete or inaccurate answers. To address these retrieval bottlenecks, we explore two strategies to improve evidence coverage and answer quality. The first is a One-SHOT retrieval method that adaptively selects chunks based on a token budget, allowing as much relevant content as possible to be included within the LLM's context window.Additionally, we design modules to further filter and refine the chunks. The second is an iterative retrieval strategy built on a Reasoning Agentic RAG framework, where a reasoning LLM dynamically issues search queries, evaluates retrieved results, and progressively refines the context over multiple turns. We identify query drift and retrieval laziness issues and further design two modules to tackle them. Through extensive experiments on a dataset of government documents, we aim to offer practical insights and guidance for real-world applications in legal and regulatory domains. The code and dataset will be released upon acceptance.

Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation

Consumer-to-consumer (C2C) marketplaces pose distinct retrieval challenges: short, ambiguous queries; noisy, user- generated listings; and strict production constraints. This pa- per reports our experiment to build a domain-aware Japanese text-embedding approach to improve the quality of search at Mercari, Japan’s largest C2C marketplace. We experimented with fine-tuning on purchase-driven query–title pairs, using role-specific prefixes to model query–item asymmetry. To meet production constraints, we apply Matryoshka Representation Learning to obtain compact, truncation-robust embed- dings. Offline evaluation on historical search logs shows consistent gains over a strong generic encoder, with particularly large improvements when replacing PCA compression with Matryoshka truncation. A manual assessment further high- lights better handling of proper nouns, marketplace-specific semantics, and term-importance alignment. Additionally, an initial online A/B test demonstrates statistically significant improvements in revenue per user and search-flow efficiency, with transaction frequency maintained. Results show that domain-aware embeddings improve relevance and efficiency at scale and form a practical foundation for richer LLM-era search experiences.

Towards Better Search with Domain-Aware Text Embeddings for C2C Marketplaces

Large Language Models (LLMs) have shown remarkable capabilities in document processing, but their inability to provide visual grounding without OCR dependencies poses significant challenges in business-critical applications. Current solutions either require model fine-tuning or rely on external OCR services, introducing additional costs, latency, and limitations in handling derived information. This paper presents ViG-LLM, a novel framework that enables closed-box LLMs to generate localization information through a multi-agent system combining U-Net-based layout deconstruction with viewport identification tasks. Evaluated on the FATURA and CORD dataset, our framework achieves perfect accuracy over spatial reasoning tuned LLM like Amazon Nova Pro, while demonstrating superior template-specific consistency. The framework maintains robust performance across LLM architectures while reducing operational costs by 60% compared to Textract-based solutions. In real-world document processing applications, the framework helps retain the high reasoning capabilities of the system in document information extraction tasks while improving explainability, reliability and human interaction for information verification. Through human-in-the-loop learning and closed-box prompt alignment techniques, ViG-LLM provides a robust, adaptable solution for visual grounding tasks in document processing workflows.

ViG-LLM: Enhancing Visual Grounding Capabilities in Closed-Box LLMs for Document Information Extraction without OCR Dependencies

Integration testing for ML systems faces unique challenges due to stochastic behavior and vast input spaces, while Continuous Integration/Continuous Delivery (CI/CD) pipelines have budgets and constraints that limit the number of tests that can be within an integration test suite. We present CAT (Coverage-Adaptive Testing), a framework that generates structured test inputs for coding-agent-based integration testing. CAT enables users to define coverage objectives through category taxonomies (e.g., adversarial attacks, compliance checks), then systematically generates high-coverage inputs through iterative refinement. A generator LLM produces candidate inputs targeting coverage gaps, while a judge LLM validates and classifies them. Greedy minimization selects compact test suites within user-specified budgets. In a case study on AWS Bedrock Guardrails, CAT achieves above 80% coverage across 27 adversarial categories for an integration test suite of 8 tests. CAT's structured outputs, natural language inputs with category labels and validation metadata, are designed to enable handoff to coding agents for test implementation. This architecture demonstrates a reusable human-AI collaboration pattern: humans define coverage objectives, CAT systematically explores the input space creating integration test inputs, and coding agents (future work) transform inputs into executable tests


CAT: Coverage-Adaptive Testing — Structured Test Suite Generation for Coding Agent Handoff

Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78× reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy–cost trade-offs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175× ($0.45–$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.

LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost

We address the challenge of training reliable code-fixing agents in real-world repositories, where complex build systems and shifting dependencies make evaluation unstable. We developed a verifiable pipeline that defines success through post-fix build validation and improves reproducibility across roughly one thousand real issues by pinning dependencies and disabling automatic upgrades. Building on this foundation, we introduced a scalable, simplified pipeline for large-scale reinforcement learning (RL). Using this setup, we supervised fine-tuned Qwen3-32B in the full pipeline and applied RL on top of the SFT model in the simplified environment. The SFT model distilled from GPT-4.1 trajectories performed comparably while being 56× smaller, and RL added 7–20% absolute gains under matched train–test conditions. “Thinking mode” achieved similar or worse results in our experiments. Both SFT and RL models failed to generalize across environments, underscoring the importance of aligning training and evaluation settings when building reliable, real-world code-fixing agents.

Agentic Reinforcement Learning for Real-World Code Repair

Rapid growth in cloud services has generated substantial demand for cybersecurity tool enhancements. Upon receiving these requests, a security assessment process is required, including investigating documentation, examining code repositories, evaluating coverage, and assigning work to tooling teams with detailed metadata. This security assessment is performed manually, consuming valuable expert time. We introduce a multi-agent GenAI system that orchestrates specialized sub-agents to automate the security assessment workflow. Evaluation on historical requests shows the system achieves over 95\% accuracy compared to human experts. Early adoption shows the system saves approximately 0.5 to 1 hour of human expert time daily, demonstrating that agent collaboration can maintain quality while reducing manual workload.

Multi-Agent Framework for Automated Cloud Security Assessment

The proliferation of foundation models has given rise to ``vibe coding," a paradigm where end-users create software by describing desired outcomes in natural language. While platforms that enable this approach are democratizing software development, they often produce simplistic, fragile applications that lack the formal guarantees required for scalable, reliable systems. The core software engineering challenges of translating ambiguous user intent into formal specifications and managing the lifecycle of AI-generated code remain largely unaddressed. This demo presents a system built on the Software Engineering for Software Makers (SE4SM) framework, which introduces a disciplined, two-pillar approach to AI-assisted development. The first pillar, Intent Engineering, employs a collaborative human-AI dialogue to transform a user's high-level concept into a set of clear, unambiguous, and verifiable specifications while considering both explicitly stated needs and implicitly inferred requirements. The second, Realization Engineering, uses a spec-driven, multi-agent system to autonomously build, test, and verify the software. This methodology preserves the accessibility of vibe coding while embedding professional engineering practices, producing more trustworthy, evolvable systems.

Next from AAAI 2026

Structured Traversal of Knowledge Graphs for Multi-hop Question Answering

Similar lecture

A Comparison of Independent and Joint Fine-tuning Strategies for Retrieval-Augmented Generation