Singapore

Human-Agent Interaction (HAI) is a methodology for augmenting human society by introducing social agents that behave with intent derived from humans. With the rapid recent advancements in generative AI, HAI is increasingly converging with AI agent research. Benchmarks in Human-Agent Interaction must evaluate not only algorithms and devices, but also the impact of individual agents&#39; behaviors and their interpersonal relationships on human society. The speaker serves as General Chair for the HAI 2025, and is a HAI Steering Committee Chair. 
From this vantage point, this talk will trace the development of HAI research over the past two decades and explore potential benchmarks for current HAI studies. Additionally, the speaker will present research findings on how dual-agent relationships influence humans, based on their own ongoing work.

AAAI 2026

How to Design Benchmarks for Augmented Society: Based on 20 Years of Trends in Human-Agent Interaction Research

human-agent interaction

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.<br><br>

To access this event page, you need to log in with the **email address you registered with**. <br>Access credentials will be sent to your email from Underline -  subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

Human-Agent Interaction (HAI) is a methodology for augmenting human society by introducing social agents that behave with intent derived from humans. With the rapid recent advancements in generative AI, HAI is increasingly converging with AI agent research. Benchmarks in Human-Agent Interaction must evaluate not only algorithms and devices, but also the impact of individual agents' behaviors and their interpersonal relationships on human society. The speaker serves as General Chair for the HAI 2025, and is a HAI Steering Committee Chair. 
From this vantage point, this talk will trace the development of HAI research over the past two decades and explore potential benchmarks for current HAI studies. Additionally, the speaker will present research findings on how dual-agent relationships influence humans, based on their own ongoing work.

invited talk

Educational assessment requires understanding student
problem-solving processes, not just final answers. Current
AI-driven analytics focus on static outcomes, missing
valuable insights from temporal dynamics. We present
Explain-from-Stroke, a practical framework that captures
invisible learning processes by integrating handwriting
dynamics with vision-language models. Our approach extracts
temporal features—writing speed, pauses, and
revisions—providing supplementary context for generating
meaningful insights into hidden aspects of student
reasoning. Deployed with real classroom data from a
Japanese secondary school, our system demonstrates 18.2\%
improvement in cognitive depth analysis over static
approaches. This work provides educators with accessible
process-oriented analysis that reveals invisible learning
processes using standard tablet technology.

Explain-from-Stroke: Capturing Invisible Learning Processes Through Handwriting Dynamics Analysis

As generative AI rapidly enters higher education, its
cognitive, motivational, and social impacts across various
disciplines remain underexplored. This qualitative study
investigates the impact of disciplinary epistemologies and
individual digital literacy on AI-assisted academic English
reading among Chinese EFL undergraduates. Guided by
Cognitive Load Theory and Self-Determination Theory, we
studied 45 students at a Hong Kong university across
hard/pure and soft/applied fields. Open-ended
questionnaires and 32 interviews were analyzed via hybrid
thematic analysis, with intercoder agreement above 85% and
member-checking. Participants in soft/applied fields more
often reported AI reducing extraneous load and supporting
deeper semantic elaboration, whereas reports in hard/pure
fields more frequently described surface-level support
(e.g., glossing terminology). Excessive reliance was
associated with cognitive offloading and an illusory sense
of mastery, shaped by digital literacy and metacognitive
awareness. Socially, AI sometimes displaced routine
exchanges but, when integrated into group contexts,
facilitated higher-order collaboration. The study
elaborates applications of CLT and SDT by showing how
disciplinary and individual factors shape AI’s cognitive
and motivational roles. Practically, it proposes
discipline-sensitive design principles, metacognitive
prompts, clear usage boundaries, and interaction-focused
affordances, pointing to deployable interventions (e.g.,
self-check tasks, staged AI fade-out scaffolds, reflective
prompts). Ethical approval and consent were obtained.

Generative AI as a Cognitive Co-Participant: Disciplinary Modulation of EFL Academic Reading Load and Motivation

In computer-supported collaborative learning environments,
analyzing student dialogue is essential for understanding
collaborative problem-solving behaviors and supporting
effective learning. Prior work often treats all dialogue
interactions uniformly, failing to capture how specific
dialogue interaction differentially impact learning
experiences and outcomes. To address this limitation, we
introduce a dialogue-based learning analytics framework
that integrates weighted temporal clustering of dialogue
with large language model-based interpretation. Our
framework identifies student interaction patterns most
predictive of group learning gains and uses these insights
to enable early prediction of learning outcomes and
generate pedagogically meaningful interpretation. We
evaluate our framework on collaborative dialogue from
middle school students engaged in a collaborative
game-based learning environment. Our results show that our
framework achieves 83.1\% accuracy in learning outcome
prediction. In addition, expert evaluations and case
studies demonstrate that the identified weighted dialogue
patterns reflect key collaborative problem-solving
behaviors recognized as important in collaborative
learning. By surfacing high-impact interaction patterns and
enabling prioritized interpretation generation, our
framework provides a promising approach for accurately
analyzing students’ collaborative dialogue.

A Dialogue-Based Learning Analytics Framework for Collaborative Game-Based Learning

Theories of embodied learning emphasize that learning
processes are grounded in bodily actions and interactions
with the environment, suggesting that movements play a
fundamental role in problem solving, decision making, and
learning. This perspective holds particular relevance for
making-based learning settings, where patterns of movement
and spatial engagement can reveal strategic expertise.
Prior research has examined distinctions between students
who learned and did not learn, but manual coding of actions
presents scalability and real-time application challenges.
To address this gap, we develop a computer vision–based
analysis pipeline for automated detection and
characterization of hand movements during complex assembly
tasks. We apply this approach to video data of students
engaged in the assembly of a differential gearbox,
quantifying metrics such as amount and speed of movement.
Results indicate that learners show fewer right-hand
movements than novices and exhibit reduced movement speed,
with a progressive decline in speed as the task unfolds.
Non-learners, by contrast, display more uneven hand
movement speed. These findings highlight measurable
differences in actions of learners and non-learners, and
therefore have implications for learning support.
Specifically, the ability to computationally distinguish
movement profiles can inform the design of adaptive
learning interventions, providing real-time performance
assessment and targeted feedback for making-based learning.

Thinking Through the Hands: An Exploratory Study of Hand Movements to Assess Students Problem-Solving in Mechanistic Reasoning Tasks

In less than a year, AI agents have evolved from a research curiosity into the foundation of some of the largest software platform updates in decades. These systems promise to automate substantial portions of knowledge work, and their progress has been rapid, with early 2025 reports by METR suggesting that the complexity of solvable tasks doubles roughly every seven months. In this talk, we take a closer empirical look at this claim by examining what it truly takes to benchmark agentic performance on long-running, open-ended knowledge work tasks. We review recent contributions from ServiceNow Research and others across domains such as browser use, multimodal understanding, data analytics, and deep research. We also discuss benchmarks that evaluate agentic safety and security, arguing that these dimensions cannot be meaningfully separated from primary task performance. Our analysis leads to a more nuanced picture of the field, highlighting both genuine advances and persistent challenges that frontier agents have yet to overcome.

Agentic Full-Stack Benchmarking for Knowledge Work

Large language models dominate current discussions on agentic AI, but their cost and complexity often limit enterprise adoption. Our research explores how small language models can be equipped with function calling, reasoning, and planning capabilities to perform effectively in enterprise workflows. Agentic AI enables systems to reason over complex states, call tools, and plan multi-step actions. I will present methods for aligning small models with these tasks, leveraging synthetic data for adaptation, and evaluating their reliability in real-world automation settings. The goal is to show how efficient models can deliver trustworthy, enterprise-grade agentic AI solutions. 
While our design principles are broadly applicable, I will focus on IT automation—drawing on IT Bench from IBM, BFCL v4, and other agentic benchmarks. IT automation is critical because enterprises depend on reliable, scalable, and secure IT services, and maintaining this reliability requires continuous monitoring, rapid incident response, and efficient remediation.

Small Language Models for Enterprise Agentic Workflows

In recent years, AI agents centered on generative AI have been rapidly gaining traction. The expansion of these AI agents goes beyond mere content generation - by automating task planning, external tool use, data collection, and result integration, they are extending AI utilization into areas of decision-making and creative work that were once the domain of humans. As a result, a more flexible and autonomous form of human–AI collaboration is emerging across diverse business and social contexts. 
At the same time, despite the growing scope of AI agent applications, the standards for evaluating their validity and reliability are not well established yet. In enterprise environments in particular, AI agents make decisions and take actions while collaborating with multiple business systems and human users, making evaluation especially challenging. There is a growing need for quantitative assessments from multiple perspectives - including agent collaboration, task success rate, and the agent’s ability to improve through self-learning and memory. 
This presentation introduces the current status and challenges of applying AI agents in enterprise settings, using Fujitsu Kozuchi AI Agent as an example. In particular, it discusses the practical difficulties of developing “domain-specific agents” revealed through real-world applications such as field work support and data analysis tasks, and outlines benchmark approaches that enable their quantitative evaluation.

Agentic AI Benchmark in Enterprise

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.

Establishing Best Practices for Building Rigorous Agentic Benchmarks

Amazon's catalog contains billions of ASINs across diverse product categories and global marketplaces, creating a large repository of structured and unstructured product data. Traditional approaches to product intelligence extraction used task-specific modeling of catalog attributes, often missing the deeper multi-modal semantic relationships within this ecosystem. Multi-modal large language models now enable reasoning about and querying product attributes at higher levels of abstraction.
This talk focuses on an agentic application developed by Amazon's Climate Pledge Friendly team for sustainability criteria validation at enterprise scale. We examine our system architecture, key learnings from production deployment, evaluation strategies, and Human-in-the-Loop workflows. Additionally, we introduce a visual prompting methodology for multi-modal LLMs that emerged from this research, accelerating the development of visual inference applications for retail product intelligence.

Agents in Retail

Membership inference attacks (MIAs) test whether a data point was part of a model's training set, posing serious privacy risks. Existing methods often depend on shadow models or heavy query access, which limits their practicality. We propose GP-MIA, an efficient and interpretable approach based on Gaussian process (GP) meta-modeling. Using post-hoc metrics such as accuracy, entropy, dataset statistics, and optional sensitivity features (e.g. gradients, NTK measures) from a single trained model, GP-MIA trains a GP classifier to distinguish members from non-members while providing calibrated uncertainty estimates. Experiments on synthetic data, real-world fraud detection data, CIFAR-10, and WikiText-2 show that GP-MIA achieves high accuracy and generalizability, offering a practical alternative to existing MIAs.

Downloads

Next from AAAI 2026

Explain-from-Stroke: Capturing Invisible Learning Processes Through Handwriting Dynamics Analysis

Similar lecture

Coverage-Constrained Human-AI Cooperation with Multiple Experts

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2026

Explain-from-Stroke: Capturing Invisible Learning Processes Through Handwriting Dynamics Analysis

Similar lecture

Coverage-Constrained Human-AI Cooperation with Multiple Experts

Downloads