United States

The Future of AI Research - Webinar Series

Artificial General Intelligence (AGI)

The AAAI 2025 presidential panel on the future of AI research aims to help all AI stakeholders navigate the recent significant transformations in AI capabilities, as well as AI research methodologies, environments, and communities. It includes 17 chapters, each covering one topic related to AI research, and sketching its history, current trends and open challenges. The study has been conducted by 25 AI researchers and supported by 15 additional contributors and 475 respondents to a community survey.

webinar

Agents and multi-agent systems (MAS) have evolved from
autonomous problem-solving entities to integrating generative
AI and LLMs, ultimately leading to cooperative AI frameworks
that enhance adaptability, scalability, and collaboration. 
**Main Takeaways**
* Multi-agent systems have evolved from rule-based autonomy to cooperative AI,
emphasizing collaboration, negotiation, and ethical alignment.
* The rise of Agentic AI, driven by LLMs, introduces new opportunities for flexible
decision-making but raises challenges in efficiency and complexity.
* Integrating cooperative AI with generative models requires balancing
adaptability, transparency, and computational feasibility in multi-agent
environments 

**Context & History** 
The field of multi-agent systems emerged in the late 1980s/early 1990s,
with its main influences coming from two disparate areas [1,2]. One was the
field of AI robotics, which had begun to seriously address the issue of integrated agent architectures: how do we assemble several cognitive components of intelligence (planning, reasoning, learning, vision,…) into an integrated computational agent? The second was the nascent area of distributed AI, which studied how multiple AI systems could be made to solve problems cooperatively, by dynamically sharing information and tasks. By the mid1990s, these ideas had given rise to the new field, driven by the vision of having (semi)autonomous AI systems – agents – working on behalf of individual users in pursuit of their users’ goals, possibly interacting with other such agents in order to do so. A key insight
was that, since delegated goals might not necessarily be in harmony, it would be necessary to equip such agents with the ability to reason socially. Thus, while AI historically emphasised components of intelligence such as reasoning and problem-solving, the new field of multi-agent systems emphasised social skills such as cooperation, coordination, argumentation and negotiation. In order to underpin those skills, the development of models of Theory of Mind became central to the field. 
By the late 1990s, the field had its own conferences and journal, and was
firmly established as a key sub-field of AI. The area flourished from the late
1990s, with enormous energy devoted to (e.g.) communication languages
for autonomous agents, protocols for cooperation, coordination, and
negotiation, and the underpinning theory of these social skills. With
respect to the latter, while in the early years the practical reasoning paradigm of AI planning had been the dominant influence on the theory of multiagent systems, by the early part of this century, game theory had become the dominant theoretical foundation. Game theory, which emerged from the field of economics, is the theory of interaction between self-interested agents. Although originally devised as a tool for studying interactions between humans and human organisations, it nevertheless seemed a natural framework for studying interactions between artificial agents. A huge body of work emerged, studying (for example), how auctions might be used to allocate scarce resources, the theory
of negotiation between self-interested artificial agents, and how agents might optimally form teams to solve problems and share the associated benefits of cooperation. Interestingly, although learning in multi-agent systems was a key component of the field from the outset, it was not the centre of attention within the field in the first decade or so. 
This initial boom period for multiagent systems lasted roughly from
the mid 1990s to around 2010-15. By the end of that time, though,
some uncomfortable questions were beginning to be asked. While the field
had generated impressive quantities of scientific results, applications seemed to be thin on the ground. For sure there were some high-profile applications. The field of security games, which emerged from multi-agent systems, used ideas from game theory to allocate scarce security resources to defend high-profile targets such as airports. This work led to deployed
applications at US airports and ports. Automated high-frequence trading
systems, which plan and execute the bulk of trades on the world’s markets,
are multi-agent systems on a global scale. And agent-based modelling,
which models socio-technical systems at the level of individual decisionmakers, received a huge boost after the 2008 financial crisis, and again after the 2020 COVID-19 pandemic, where it was demonstrated to be an important tool for modelling the spread of contagion: financial in the first case; epidemiological and social policy in the second. But for all these successes, the core vision of multi-agent systems - where agents function in the context of other agents is an active area of research, exploring social concepts such as norms, organisations, practices and also values, is ongoing work, but for a large part outside the AAMAS community, but within social simulation research, and with results informing and
shaping policy-making in several areas from public health to transportation,
and urban transformation. 
While applications of multi-agent systems research (AI agents interacting
with other AI agents) has not, as yet, lived up to early expectations,
individual dialogue agents such as Alexa, Siri, Cortana are now an everyday
reality, and trace their historical roots both to work on intelligent agents
in the 1990s and work from the NLP community on dialog systems. Many
other applications of this thread of work have achieved success over the past three decades: automated call center assistants, customer service assistants, smartphone virtual assistants, smart speaker assistants, home robot assistants, that can converse with human users and accomplish tasks like ticket booking, restaurant reservations, online shopping, medical and health assistance, and sales assistance, empowered by AI modules like
speech recognition, natural language understanding, dialog management (i.e. state tracking and dialog policy), and natural language generation and speech synthesis. 
As ML surged in the early part of this century, activity in this area increased
within the multi-agent systems field. Multi-agent reinforcement
learning (MARL) grew to become the single biggest area within the field,
possibly driven in part by the fact that developing MARL experiments can be done relatively quickly and without recourse to expensive hardware. At the time of writing, while MARL represents a significant sub-field of ML as a whole, it seems to lack any clear unifying vision or direction – or application. 
**Current State & Trends** 
The emergence of LLMs from 2020 onwards has also led to increased
interest in agents [3]. LLMs can be used as part of a workflow to
automate routine tasks, and the general capabilities of such “agents” for
planning and problem solving is widely discussed. In this context, the concept of Agentic AI refers to the integration of generative AI and LLMs into autonomous agent frameworks aiming to leverage the generative capabilities of such models to enhance interaction, creativity, and real-time decision making in dynamic environments. As we write this (late 2024) there has been an explosion of startup companies hoping to commercialise such agents. Despite this renewed enthusiasm, the original aims of AAMAS from 30 years ago, such as building robust, autonomous multiagent systems capable of complex coordination and long-term reasoning, have not been fully realized. The extent to which this new wave of agent activity is informed by what went before is unclear. 
The challenge now is to understand what multi-agent systems mean in
the era of LLMs. The current direction of agentifying LLMs may lead to
overly complex and unnecessary architectures and heavy computational
costs, whereas adopting a multi-agent paradigm to the development and use of LLMs may offer a sustainable way to compose, diversify, and integrate approaches effectively. Even though distribution was one of the original drivers for the MAS field, this is still a largely unexplored direction under the current paradigm. Another trend nowadays is to recover ideas from classical cognitive architectures to add common sense skills to autonomous agents. 
An emerging trend is multi-agent architectures, which structure
AI components into modular systems that improve transparency,
adaptability, and ethical alignment. The focus on cooperative agents
highlights a shift toward AI that prioritizes collaboration, negotiation, and shared decision-making. By applying modularity, encapsulation,
and separation of concerns, these architectures enable scalable teamwork
between autonomous agents and humans, making them ideal for
hybrid AI applications requiring trust, explainability, and domain-specific
expertise. 
**Research Challenges** 
* Identify challenges and benefits of embedding GenAI-driven agents
into MAS, focusing on enhancing collaboration without disrupting
existing dynamics.
* Investigate how LLM-powered agents can improve negotiation and
decision-making in dynamic multiagent environments while ensuring
ethical alignment and safety.
* Develop architectures that integrate LLM-driven agents while maintaining
scalability, transparency, and computational efficiency in multiagent settings. 
**Community Opinion** 
The survey responses indicate a
majority of respondents finding this
theme relevant to their research, with
a growing interest in integrating Large
Language Models (LLMs) into multiagent systems. Many participants
already use AI agents, with LLMs being
the most common technique (29.34%),
highlighting their expanding role in AI driven applications. 
The potential of multi-agent systems
leveraging LLMs is seen in areas such
as collaborative problem-solving
(68.86%), distributed decision-making
(54.49%), and social simulations
(41.32%). However, challenges persist,
including misalignment between LLMs’
general knowledge and specific system
needs (59.88%), lack of interpretability
(59.28%), and security risks (50.90%).
These concerns suggest a need for
improved explainability, alignment
strategies, and robust security measures
to ensure effective deployment. 
There is also a debate on the necessity
of agentifying LLMs—while 51.5%
believe multi-agent LLM paradigms
are essential for sustainable AI,
42.33% disagree that they introduce
unnecessary complexity. The
computational cost-benefit balance
remains uncertain, with responses
divided on whether LLMs outweigh
their costs. 
The textual responses highlight a broad
spectrum of perspectives on integrating
Large Language Models (LLMs) into
multi-agent systems (MAS), with some
advocating for hybrid approaches
rather than relying solely on LLMs. Many
respondents stressed the need for
diverse AI architectures, emphasizing
modular, multi-technology systems
where LLMs play a role but do not
dominate. Governance, coordination,
and adaptability emerge as key
advantages of MAS, while concerns
include increased complexity, lack
of theoretical guarantees, and high
computational costs. Several responses
criticize the overemphasis on LLMs,
questioning whether they are truly
essential or merely a current trend.
Others highlight practical challenges
such as grounding, alignment, and
robust communication protocols,
pointing out the need for new
frameworks that integrate symbolic
reasoning, structured governance,
and scalable architectures. Overall,
the discussion reflects a critical but
open stance toward agentifying LLMs,
suggesting that context, application
domain, and technological diversity will
shape their effectiveness in multi-agent
environments. 
In summary, the survey reflects
optimism about LLM-driven multiagent systems, but also underscores
the need for addressing key challenges
before widespread adoption.
 
1. Yoav Shoham. Agent-oriented programming. Artificial Intelligence. Artificial Intelligence. 60 (1): 51–92.
2. Michael Wooldridge. An Introduction to Multi-agent Systems 2e. Wiley, 2009.
3. Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic. Agents. Google whitepaper. https://archive.org/details/google-ai-agents-whitepaper

AI agents

The ability to reason has been a salient characteristic of human intelligence, and there is a critical need for verifiable reasoning in AI systems. 
**Main Takeaways** 
* Reasoning has always been seen as a core characteristic of human intelligence.
Reasoning is used to derive new information from given base knowledge; this
new information is guaranteed correct when sound formal reasoning is used,
otherwise it is merely plausible.
* AI research has led to a range of automated reasoning techniques. These
reasoning techniques have given rise to AI algorithms and systems, including
SAT, SMT, and constraints solvers as well as probabilistic graphical models, all of
which play a key role in critical real-world applications.
* While large pre-trained systems (such as LLMs) have made impressive
advancements in their reasoning capabilities, more research is needed to
guarantee correctness and depth of the reasoning performed by them; such
guarantees are particularly important for autonomously operating AI agents 

**Context & History** 
Reasoning is a core component of
human intelligence. From the dawn
of humanity, abductive reasoning
has been used to predict danger and
inductive reasoning made it possible
to learn regularities governing the
world. Beginning in Ancient Greece,
deductive reasoning techniques were
developed to draw valid conclusions
that follow logically from premises
known to be true. The development
of reasoning methods with such a
priori guarantees was a key factor in
the advancement of modern science,
mathematics, and engineering; notably,
according to philosophers such as
Charles Sanders Peirce, the interplay
between abduction, deduction, and
induction forms the basis of the
scientific method and hence all modern
science. Attempts to mechanize
logical reasoning can be traced back to
13th-century philosopher Ramon Lull
and lie at the heart of the concept of
computation. Probabilistic reasoning
and inference have also profoundly
impacted reasoning, often relying on
the celebrated theorem by Thomas
Bayes on inverse probability that also
forms the basis for many machine
learning and statistics approaches.
Finally, the evaluation of correct (sound)
reasoning lies at the heart of most
quantitative assessments of human
cognition. 
Not surprisingly, reasoning has been
central to the AI enterprise. Indeed,
the earliest research in AI – from Logic
Theorist onwards [1] – had a strong
focus on reasoning [2]. Since the 1960s,
AI has also embraced probabilistic
reasoning and models, initially for
medical diagnosis [3]. Since then,
the reasoning tasks addressed in AI 
research have covered the gamut from
planning and temporal reasoning to
diagnosis and explanation. While early
AI has paid attention to both plausible
reasoning (case-based, analogical,
qualitative) and sound formal reasoning
with guarantees (logical, probabilistic,
constraint-based), over the years,
the focus has shifted more towards
reasoning with formal guarantees.
There are good reasons for this when
designing AI systems and techniques
that compensate for human limitations
and weaknesses since reasoning
with guarantees is challenging for
humans. This has led to practically
impactful applications of AI systems
such as SAT, SMT, and constraints
solvers, including the verification of
correctness properties of computer
hardware and software, the safety of
communications protocols, the design
of new proteins, and, more recently, the
robustness of neural networks against
adversarial attacks. It has also resulted
in probabilistic graphical models [4,
5], which are powerful modeling and
inference tools that have found their
way into numerous applications of
reasoning in medicine, robotics, and
beyond. 
**Current State & Trends** 
The emergence of the Internet and the
associated technology that made it
possible to capture the human digital
footprint at scale, as well as the leaps in
computing power, have made possible
novel approaches to learning bottom up from data. Of particular interest are
large pre-trained models, such as LLMs,
that have shown surprising abilities in
plausible reasoning. Unlike the earlier
research on reasoning in AI, LLMs
have focused on plausible reasoning patterns as they emerge automatically
after large-scale training on petabyte
corpora. While the results have been
quite remarkable so far, the reasoning in
this context has been of the “plausible”
variety with no guarantees. 
Meanwhile, sound formal reasoning
techniques remain key to important
and impactful applications of cuttingedge AI technology for the verification
of computer hardware and software,
as well as for real-world planning and
resource allocation problems. They
are also increasingly recognized as a
crucial basis for the formal verification
of machine learning techniques such
as neural networks, e.g., in the context
of local robustness against adversarial
attacks [6]. Significant research activity
takes place in these areas, focusing on
improving various types of reasoning
algorithms (notably with respect to their
computational complexity), leveraging
learning within sound formal reasoning,
and combining reasoning and learning
techniques [7, 8]. 
**Research Challenges** 
Bringing some of the rigorous a priori or
post hoc guarantees back into plausible
reasoning patterns turbocharged by
the pre-trained models has become an
active and promising area of research
– especially where AI systems need to
work autonomously in safety-critical
domains. Research on so-called “large
reasoning models” as well as on neurosymbolic approaches is addressing
these challenges. 
Furthermore, even though formal
reasoning with correctness guarantees
is currently considerably less in vogue
than the use of generative AI techniques
for plausible reasoning, formidable and essential challenges also remain in that
area. In this context, the combination
of machine learning techniques with
formal reasoning techniques holds
considerable promise for economically
and socially valuable breakthroughs,
notably in the area of AI safety and
transparency.
The questions and challenges we face
range from the philosophical:
* What exactly is “reasoning”?

to the practical:
* Can LLM ‘reasoning’ be trusted?

and include: 
What does the future hold for the
advancement and role of symbolic
reasoning?
* To what extent can LLMs or other
generative models reproduce or
replace symbolic reasoning?
* To what degree will symbolic
reasoning be necessary or sufficient
to overcome the current limitations
of LLMs?
* How well can AI reasoning,
especially LLM ‘reasoning,’ be
explained and understood?
* How can computers better
understand and simulate human
reasoning?
* What is the role of collaborative
reasoning between humans and
computers?
* How best can LLMs and symbolic
reasoning be integrated into “neurosymbolic reasoning”?
* Are further breakthroughs, beyond
both LLMs and traditional symbolic
reasoning, required to achieve AGIlevel reasoning?
* What forms of reasoning can best
support humans when dealing with
various challenges, e.g., in medical,
scientific, engineering, and legal
domains? 

**Community Opinion** 
The AAAI community appears to
strongly agree on the importance
of reasoning in AI systems. In our
community survey, slightly over 55%
of the respondents chose to answer
specific questions related to the topic
of reasoning. Of these, 79% indicated
that the topic of reasoning is relevant
to their research (with 44.7% marking
it as “very relevant”). Of the properties
required for referring to a process
as reasoning, 77.5% of the survey
participants marked “Knowledge can
be incorporated”, 72.5% “Explanations
can be provided,” and 56.9% “Involves
multiple steps to arrive at a conclusion”.
Interestingly, merely 37.4% indicated
“Guaranteed correctness of inference
results/outcomes”, and only 23.7% that
“A formal system and solver is used,”
which reflects the recent focus on
informal, plausible reasoning, likely in
the context of generative AI methods.
This suggests that an effort may be
warranted to better communicate the
importance and success of formal,
sound reasoning techniques. Finally,
44.7% of respondents agreed that
“Reasoning involves a search process.” 
There was broad agreement among
survey participants that focusing
reasoning research in AI on humanlevel reasoning is valuable (41.6%) or
even essential (47%); similarly, a focus
on domain-specific reasoning abilities
was seen by 49.6% of respondents as
valuable, and by 42.8% as essential.
This clearly reflects the importance
attributed to a research focus on
reasoning. 
The community also sees an exciting
potential of synergy offered by logical
and probabilistic models of reasoning
that were developed in AI prior to large
pre-trained models. This is clearly
reflected in the fact that 76.9% of survey
participants marked the integration
of learning and reasoning approaches
as very important (6 or 7 on a scale
of 7); interestingly, the percentage
of respondents that considered
Explainability and verifiability as very
important was similarly high (at 71.7%). 
Finally, 61.8% of survey participants
estimated the minimal percentage of
symbolic AI techniques required for
reaching human-level reasoning to be
at least 50% (with 24.8% estimating it
at 75% or more, compared to 38.2%
estimating it at 25% or below). What
remains unclear is the degree to
which AI researchers and practitioners
realize that decidedly superhuman
levels of reasoning are required for
and displayed in the prominent and
successful applications of formal AI
reasoning techniques for scientific
and mathematical discovery and
engineering applications, as well as in AI
safety. 
1. Newell, A. & Simon, H. (1956). The logic theory machine: A complex information processing system. IRE Transactions on Information Theory 2: 61-79. 
2. Brachman, R. and Levesque, H. (2004) Knowledge Representation and Reasoning (1st Ed). Morgan Kaufman. 
3. Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach (4th Ed). Pearson. 
4. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman. 
5. Koller, D. and Friedmann, N. (2009) Probabilistic Graphical Models. The MIT Press. 
6. König, M. et al. (2024) Critically Assessing the State of the Art in Neural Network Verification. Journal of Machine Learning Research 25(12): 1-53 
7. Guo, D. et. al. (2025) DeepSeek-R: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948 
8. Kambhampati, S. (2024). Can Large Language Models Reason and Plan? Annals of New York Academy of Sciences. March 2024.

AI Reasoning

Improving factually and trustworthiness of AI systems is the
single largest topic of AI research today, and while significant
progress has been made, most scientists are pessimistic that
the problems will be solved in the near future. 
**Main Takeaways** 
* An AI system is factual if it refrains from outputting false statements. Improving
factually of AI systems based on neural-network large language models is
arguably the biggest area of AI research today.
* Trustworthiness extends trustworthiness to include criteria such as human
understandability, robustness, and the incorporation of human values. Lack of
trustworthiness has always been an obstacle for deploying AI systems in critical
applications.
* Approaches to improving the factuality and trustworthiness of AI systems
include fine-tuning, retrieval-augmented generation, verification of machine
outputs, and replacing complex models with simple understandable models. 

**Context & History** 
A factual AI system does not output
erroneous information or hallucinate
answers. Before the era of generative
AI, problems with factuality arose when
systems were trained on bad data,
as captured by the slogan, “garbage
in, garbage out”. Work on methods
for improving data quality has a long
history in AI [1]. 
Generative AI, and in particular
large language models, employ
reconstructive memory - that is, they
rebuild memories as needed on the
basis of distributed bits of information
rather than retrieve memories from
a fixed store. The earliest generative
LLMs made an impact with their ability
to generate coherent but entirely
imaginary stories [2]. Factuality of LLMs
on a given domain was improved by
fine-tuning the model on domain data
[3]. 
Trustworthiness is a broader concept
than factuality because it includes
criteria such as understandability,
robustness, and respect of human
values. A traditional approach for
improving understandability of AI
systems is to replace complex blackbox models with simple humanunderstandable models - such as
naive Bayes [4] or generalized linear
regression [5]. Research on robustness
of machine learning studies how the
outputs of a model vary with small
changes in its training data. For
example, contrastive learning is a
method to train deep neural nets with
increased robustness [6]. Further
discussion of robustness in generative
AI appears in
this report’s section on Reasoning.
Discussion of respect of human values
by AI systems appears in many other
sections of this report and so will not be
discussed here. 

**Current State & Trends** 
As noted, fine-tuning remains the
main approach used by scientists
and engineers to improve factuality
of generative AI systems. In addition
to fine-tuning on domain-specific
documents, modern fine-tuning
includes reinforcement learning with
human feedback from thousands of
people. The cost of employing such
large numbers of human evaluators
is a major bottleneck for scaling AI
systems, and so there is much interest
in discovering methods to reduce the
amount of human feedback needed [7]. 
The second main technique for
improving factuality of generative AI
is retrieval-augmented generation
(RAG) [8]. In response to a question,
the system gathers a set of relevant
documents using traditional
information retrieval algorithms. The
AI system is then prompted to generate
an answer by combing through and
summarizing the retrieved documents.
While RAG can improve factuality, it
is dependent on the quality of data
retrieved. For example, if the target
document set is the entire web, it
can end up incorporating incorrect
information and even satirical stories in
its answer. 
Related to RAG is enabling the
generative AI system to use tools for
fact checking. Tools used by generative
AI systems include calculators, factual
databases such as citation indexes, and
formal planning and reasoning systems
[9]. A recent approach to improving
factuality is to provide the system with
a set of rules that state constraints on
space of answers. The output from the
model is verified against these rules and
inconsistent responses are culled [10].
Amazon Web Services already supports
this approach with “automated
reasoning checks” [11]. 
A third technique for improving
factuality of generative AI is chainof-thought (CoT), where a series of
prompts breaks down a question into
smaller units [12]. CoT often includes
steps where the model is asked to
reflect back on its tentative conclusions
and see if any are hallucinations. CoT
is discussed in more detail in the
Reasoning section of this report. 
The impact of data quality on factuality
was mentioned above. In addition
to fine-tuning on human curated
data, there is recent work on creating
synthetic data that is guaranteed to be
high quality for fine-tuning [13]. 
Trustworthiness, we noted, generalizes
factually and includes understandability
and robustness. One approach to
making neural network models more
understandable is to factor them into a
set of recognizers for high level features
and then combine the features using an
understandable model such as additive
regression [15]. Another approach is to
tease out how concepts and rules are
actually represented in a trained [16].
Understandability can also be
improved by employing CoT techniques
to ask a generative AI system to explain
the steps in its reasoning [17] or tell
the user when the system is uncertain
about a conclusion [18]. Finally, a
generative AI system can be asked not
to output a single answer, but instead
to distill a complex set of information into a simple human understandable
representation such as a decision tree
[19]. 
**Research Challenges** 
Factuality is far from solved. There
are a growing number of benchmark
dataset designed to test the factuality
of LLMs. One of the latest, SimpleQA
from Google, is a collection of simple,
unambiguous, timeless, and challenging
factual questions and answers
[14]. As of December 2024, the best
models from OpenAI and Anthropic
correctly answered less than half of the
questions. 
Robustness in generative AI can
be improved, as noted above, by
employing robust loss functions such
as contrastive learning. Adversarial
training, which applies perturbations in
the embedding space during training,
can improve both robustness and
generalization [20]. In addition, the
techniques for factuality generally
improve robustness as well. 
**Community Opinion** 
Over 75% of the AAAI community
strongly agreed that factuality and
trustworthiness were relevant or very
relevant to their own research. 
All six approaches mentioned in
the survey for improving factuality - external fact-checking tools,
reinforcement, improving data quality,
data curation, synthetic training, and
new neural net architectures - found
support. The greatest demand was
for more research on new neural net
architectures (73% marked important
or very important), followed closely by
external fact-checking tools (70%). 
For trustworthiness, new neural
net architectures were also viewed
as most important (77% important
or very important), followed by
enabling models to describe their
reasoning processes (70%) and use
of understandable models instead of
neural networks (61%). Notably, few
viewed research on giving AI systems
human-like personalities as important
to improving trustworthiness (24%).
Finally, the community mostly agreed
(59%) that trustworthiness as currently
formulated is ill-defined. Most also
disagreed (around 60%) that either
factuality or trustworthiness would
soon be solved. 
The AAAI community suggested
a number of additional aspects of
factuality and trustworthiness that were
not covered above. These included:
* The ability to understand and
present different sides of the same
issues, including pros and cons for
each one.
* Understanding that trustworthiness
depends upon the context of the
domain, organizational objectives,
and user objectives. It is wrong
to think of an AI system as simply
being trustworthy or untrustworthy
without regard to context.
* Transparency needs to go beyond
the models used to the actual
sources of training data. This should
include multi-source verification of
facts ingested by the models.
* The focus of work should be
on risk and mitigation rather
than on “solving” factuality and
trustworthiness.
* Attention needs to paid to giving
AI agents the ability to update
their knowledge while maintaining
reliability.
 

1. Budach, Lukas, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Hazar Harmouch and Felix Naumann (2022). The Effects of Data Quality on Machine
Learning Performance. https://arxiv.org/pdf/2207.14529
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. Retrieved from https://cdn.openai.com/betterlanguage-models/language_models_are_unsupervised_multitask_learners.pdf
3. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.48550/
arXiv.1810.04805
4. Pedro Domingos & Michael Pazzani (1997). On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Learning, 29(2-3), 103–130. https://doi.
org/10.1023/A:1007413511361
5. Caruana, Rich, Yin Lou, Johannes Gehrke, Paul Koch, M. Sturm and Noémie Elhadad (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day
Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015). https://people.dbmi.columbia.edu/noemie/
papers/15kdd.pdf
6. Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR 2006), 2, 1735–1742. https://doi.org/10.1109/CVPR.2006.100
7. Hu, C., Hu, Y., Cao, H., Xiao, T., & Zhu, J. (2024). Teaching language models to self-improve by learning from language feedback. Findings of the Association for Computational Linguistics
(ACL 2024). Retrieved from https://aclanthology.org/2024.findings-acl.364/
8. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for
knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33, 9459–9474. Retrieved from https://arxiv.org/abs/2005.11401
9. Guan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023). Leveraging pre-trained large language models to construct and utilize world models for model-based task
planning. Proceedings of the 33rd International Conference on Automated Planning and Scheduling (ICAPS 2023). Retrieved from https://arxiv.org/abs/2305.14909
10. Backes, J., Bolignano, P., Cook, B., Dodge, C., Gacek, A., Luckow, K., Rungta, N., Tkachuk, O., & Varming, C. (2018). Semantic-based automated reasoning for AWS access policies using
SMT. In 2018 Formal Methods in Computer-Aided Design (FMCAD) (pp. 1–9). IEEE. https://doi.org/10.23919/FMCAD.2018.8602994
11. Barth, Antje (2024). Prevent factual errors from LLM hallucinations with mathematically sound Automated Reasoning checks (preview). Posted 3 Dec 2024, retrieved 8 Feb 2025. AWS
News Blog, permalink https://aws.amazon.com/blogs/aws/prevent-factual-errors-from-llm-hallucinations-with-mathematically-sound-automated-reasoning-checks-preview
12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in
Neural Information Processing Systems (Vol. 35, pp. 24824–24837). https://proceedings.neurips.cc/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
13. Ding, Bosheng, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu and Shafiq R. Joty (2024). Data Augmentation using LLMs: Data
Perspectives, Learning Paradigms and Challenges. Annual Meeting of the Association for Computational Linguistics (2024). https://aclanthology.org/2024.findings-acl.97.pdf
14. Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus (2024). Measuring short-form factuality in large language
models. https://doi.org/10.48550/arXiv.2411.04368
15. Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., & Hinton, G. E. (2021). Neural additive models: Interpretable machine learning with neural nets. Advances in
Neural Information Processing Systems, 34, 2021. https://proceedings.neurips.cc/paper/2021/hash/251bd0442dfcc53b5a761e050f8022b8-Abstract.html?utm_source=chatgpt.com
16. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin,
A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., & Henighan, T. (2024). Scaling Monosemanticity: Extracting
Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
17. Yeo, W. J., Ng, X. X., Le, T. K. C., & Lu, X. (2024). How interpretable are reasoning explanations from prompting large language models? Findings of the Association for Computational
Linguistics: NAACL 2024. Retrieved from https://aclanthology.org/2024.findings-naacl.138
18. Lin, S., Hilton, J., & Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research. https://openreview.net/pdf?id=8s8K2UZGTZ
19. Chen, Y., Zhang, L., Wang, H., & Li, J. (2025). Zero-Shot Decision Tree Construction via Large Language Models. arXiv preprint arXiv:2501.16247. Retrieved from https://arxiv.org/
abs/2501.16247
20.Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020). Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994. Retrieved from https://
arxiv.org/abs/2004.08994

AI Factuality & Trustworthiness

How should we challenge exaggerated claims about AI’s
capabilities and set realistic expectations?

**Main Takeaways**

* Over the last 70 years, against a background of constant delivery of new and important technologies, many AI innovations have generated excessive hype.
* Like other technologies these hype trends have followed the general Gartner Hype Cycle characterization.
* The current Generative AI Hype Cycle is the first introduction to AI for perhaps the majority of people in the world and they do not have the tools to gauge the validity of many claims 

**Context & History**

Artificial intelligence, or AI, is the field
that studies the synthesis and analysis
of computational agents that act
intelligently [6]. AI has gone through
hype cycles multiple times since the
1956 workshop that established the
name AI and set the course for early
computer science departments to
include AI as a major component of
their research and teaching. All hype
bubbles eventually burst, as the
essence of hype is that it is beyond
reality. Over the decades this has led to
AI winters where funding has dried up
for all of AI or for specific aspects of AI
such as neural networks or robotics.

A study published 2017 on trends with
the public perception of AI over a 30-
year period found that discussion about
AI had sharply increased since 2009
and that discussions in the public press
had been consistently more optimistic
than pessimistic [4]. The study also
found that hopes about AI applications
of AI in healthcare and education
were increasing over time. Another
finding was that concerns were growing
about loss of control of AI, ethical
implications, and the negative impact of
AI on work. 
Perhaps the difference in recent years
with prior periods is that hype has
gone beyond the pages of academic
conferences, conference papers, and
scientific magazines, out into both the
mainstream media, and social media.
AI and Artificial Intelligence have
become common words that nontechnical people have heard about
and a common subject for leaders
of almost all countries to talk about.
Governments, for the first time, have AI
policies. 
One of the problems is that AI is
actually a wide-reaching term that can
be used in many different ways. But
now in common parlance it is used
as if it refers to a single thing. In their
2024 book [5] Narayanan and Kapoor
likened it to the language of transport
having only one noun, ‘vehicle’, say, to
refer to bicycles, skate boards, nuclear
submarines, rockets, automobiles, 18
wheeled trucks, container ships, etc.
It is impossible to say almost anything
about ‘vehicles’ and their capabilities
in those circumstances, as anything
one says will be true for only a small
fraction of all ‘vehicles’. This lack of
distinction compounds the problem
of hype, as particular statements get
overgeneralized. 
The hype also sets expectations for
ordinary people. Many are fearful that
they will lose their jobs to AI in the
short term. Social scientists then work
to solve labor disruptions, e.g., for
displaced truck drivers [6], based on
predictions about AI (and in this case,
self-driving trucks) and its adoption that
turn out to be wildly optimistic. There
are no deployed self-driving trucks in
the predicted time frame. 
Hype in response to a technology
trigger is not restricted to AI. Indeed
the business intelligence company
Gartner, has deliberately made
a practice of using a graphical
representation of hype levels through
five stages that are common for many
technologies: (1) technology trigger,
(2) peak of inflated expectations, (3)
trough of disillusionment, (4) slope
of enlightenment, and (5) plateau
of productivity. They have used this
framework to track many technologies,
including quantum computers, block
chain, autonomous vehicles, nanotechnology, etc. In November 2024 they
[1] estimated that hype for Generative
AI had just passed its peak and was on
the downswing. 
The question for AI professionals is
how to respond to this hype, how to
question it, and how to help others
understand what is hubris, while
maintaining their own intellectual
modesty and probity. This is hard to do
in the middle of outsized claims about
one’s own field, and often it is up to
future historians to carefully dissect
past scientific arguments. 
Historian Thomas Haigh has tried to
do such a dissection, almost in real
time, in a recent series of articles in the
Communications of the ACM. In [2] he
gives a post-mortem on the impact of
over-hype in AI that resulted in what is
known as the AI-winter in the 1980s. His
one line summary is: “Fallout from an
exploding bubble of hype triggered the
real AI Winter in the late 1980s.” In [3] he
makes a comparison between the hype
of today and of those earlier times.
He summarizes this particular opinion
piece with the line: “From engines of
logic to engines of bullshit?” 
**Research Challenges** 
Many of us who have worked in AI for
decades face the challenge of trying to
remain honest brokers when we see
that many of the public statements of
people quite new to the field are out of
line with reality. 
The big question is whether, given
the dynamics of social media and the
search for clicks, professional opinions
and peer reviewed research papers
have any impact on dampening the
overclaims and the ways they distort
common understanding of where AI is,
and what is its potential in one year, five
years, ten years, etc. 
If we are currently left out of the
conversations how can we change that? 
**Community Opinion** 
The Community Survey gives
perspectives on the reactions to the AI
Perception vs Reality theme. First, the
results of the survey are summarized
here. 36% of the survey respondents
chose to answer the questions for this
theme. This is the summary breakdown
of the responses to each question: 
**1. How relevant is this Theme for your
own research?** 72% of respondents
said it was somewhat relevant (24%),
relevant (29%) or very relevant (19%). 
**2. The current perception of AI
capabilities matches the reality of AI
research and development.** 79% of
respondents disagreed (47%) or strongly
disagreed (32%). 
**3. In what way is the mismatch
hindering AI research?** 90% of
respondents agreed that it is hindering
research: 74% agreeing that the
directions of AI research are driven by
the hype, 12% saying that theoretical AI
research is suffering as a result, and 4%
saying that less students are interested
in academic research. 
**4. Should there be a communitydriven initiative to counter the hype
by fact-checking claims about AI?** 78%
yes; 51% agree and 27% strongly agree. 
**5. Should there be a communitydriven initiative to organize public
debates on AI perception vs reality,
with video recordings to be made
available to all?** 74% yes; 46% agree
and 28% strongly agree. 
**6. Should there be a communitydriven initiative to build and maintain
a repository of predictions about
future AI’s capabilities, to be checked
regularly for validating their accuracy?**
59% yes; 40% agree and 29% strongly
agree. 
**7. Should there be a communitydriven initiative to educate the public
(including the press and the VCs) about
the diversity of AI techniques and
research areas?** 87% yes; 45% agree
and 42% strongly agree. 
**8. Should there be a communitydriven initiative to develop a method
to produce an annual rating of the
maturity of the AI technology for
several tasks?** 61% yes; 42% agree and
19% strongly agree. 
Since the respondents to this theme
are self-selected (about a third of all
respondents), that bias must be kept
in mind. Of those who responded,
a strong and consistent (though not
completely monolithic) portion felt that
the current perception of AI capabilities
was overblown, that it had a real impact
on the field, and that the field should
find a way to educate people about the
realities.

1. Chandrasekaran, A. [2024]. What’s Driving the Hype Cycle for Generative AI, 2024. https://www.gartner.com/en/articles/hype-cycle-for-genai
2. Haigh, T. [2024]. How the AI Boom Went Bust. Vol. 67 No. 2, pp 22-26.
3. Haigh, T. [2025]. Artificial Intelligence Then and Now. Vol. 68 No. 2, pp 24-29.
4. Fast, E. and Horvitz, E. [2017]. Long-term trends in the public perception of artificial intelligence, AAAI’17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
February 4, 2017, pp 963-969. https://ojs.aaai.org/index.php/AAAI/article/view/10635
5. Narayanan, A. and Kapoor, S. [2024]. AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference. Princeton University Press.
6. Poole, D. L. and Mackworth, A. K. [2023]. Artificial Intelligence: Foundations of Computational Agents, 3rd edition. Cambridge University Press.
7. Wang, S., Mack, E. A., Van Fossen, J. A., Medwid, M., Cotten, S. R., Chang, C. H., Mann, J., Miller, S. R., Savolainen, P.T. and Baker, N. [2023]. Assessing alternative occupations for truck
drivers in an emerging era of autonomous vehicles. Transportation Research Interdisciplinary Perspectives, Vol 19. May. https://doi.org/10.1016/j.trip.2023.100793

AI Perception vs. Reality

It is important to encourage and support research on a
variety of AI paradigms, old and new. This includes diverse
methodologies (beyond just neural networks) both new and
old, interdisciplinary collaboration, and consideration of
societal implications. 
**Main Takeaways**
* Historically, the field of AI has simultaneously encompassed many different methodologies and research paradigms.
* There is a risk that the current convergence of the field towards focusing on neural approaches could impede innovation.
* We call for active support of research on classical (non-neural) approaches, as well as research that combines neural approaches with other approaches, and that integrates various paradigms into more complete cognitive architectures. And we especially encourage support of creative investigations of completely new paradigms that may be the key to overcoming the limitations of existing paradigms. 

**Context & History** 
There’s a long history in the field of AI
of separate subcommunities deeply
pursuing different approaches to
replicating intelligence in computers.
Sometimes they have been organized
around approaches, such as Planning,
Evolutionary Computation, Constraint
Satisfaction or Combinatorial Search.
Sometimes they have been organized
around applications, such as Computer
Vision, Natural Language Processing, or
Robotics. 
While there are usually some areas
that are more “in fashion” than
others, a few notable area disputes
notwithstanding, the community has
generally been good about tolerating,
and even encouraging, a diversity of
approaches. Indeed, it could be argued
that the current flourishing of neural networks-based generative AI is a result
of this tolerance. Neural networks were
introduced and studied even before
the term “Artificial Intelligence” was
coined in the 1950s, and there was a lot
of research on the topic in the 1960s.
After they didn’t live up to the level of
hype about them at the time, there
was a time when the majority of the
field considered “connectionism” to
be a dead end. But a subcommunity
persisted, and eventually got their day
in the sun (to say the least). 
**Current State & Trends** 
However, there is now a risk that this
tradition of diversity will be lost. It is
likely that one or more of the currently
unfashionable research areas could
eventually also see its day in the sun.
But we do not currently know which
ones. Due to the current dominance
of neural approaches, many of the
other approaches are losing steam,
or even being redefined as no longer
AI (A recent article in IEEE Spectrum
implied in its headline that classical
search is not AI: https://spectrum.
ieee.org/chip-design-ai). Indeed, as a
community, we seem to be in danger of
discouraging newcomers from pursuing
any alternatives. 
We think that would be a mistake. On
the contrary, for the long-term health of
the field, it is important that we find a
way to support the brave souls who are
resisting jumping on the bandwagon,
even though their papers may be less
likely to be accepted, and/or may
accumulate fewer citations. Some of
those papers may nonetheless end up
being immensely impactful. And even
(or especially) for people focussing their
attention on neural networks, we find it
important that they are knowledgeable
about alternative paradigms so as not
to impede innovation by needing to
“reinvent the wheel”. 
This isn’t to say that we condone
ignoring progress in neural-networkbased generative AI. It may be the
biggest revolution our field has seen,
and deserved an enormous amount of
attention. Just not all the attention. 
**Research Challenges** 
We predict that some of the future
breakthroughs will come from other
areas, either on their own, or in
combination with neural and other
classical methods. 
For example, investigations of the
planning capabilities of Large Language
Models find that they are really
unable to reason and plan effectively
[Valmeekam et al., 2023; Valmeekam
et al., 2024]. It may be that some
form of symbolic reasoning system
is required to work with the LLM to
produce sensible plans. Neurosymbolic
approaches are moving in that
direction. And, similarly, conformal
prediction [Angelopoulos and Bates,
2023] is an effort to inject probabilistic
reasoning into neural models. 
We call on the AI community to
complement its deep focus on
the capabilities and limitations
of neural approaches by actively
supporting research on classical
(non-neural) approaches such as
search, optimization, constraint
satisfaction, and causal reasoning,
as well as research that combines
neural approaches with symbolic and
probabilistic approaches, and that
integrates various paradigms into more
complete cognitive architectures. And
we especially encourage support of
creative investigations of completely
new paradigms that may be the key to
overcoming the limitations of existing
paradigms. 
This support could come in the
form of workshops devoted to such
investigations, and should also include
directing some funding towards
these priorities. We should especially
seek ways to encourage and support
researchers, old and new, who are
interested in pursuing new ideas
from new perspectives, as well as
opportunities for intersecting various
new and existing approaches, even
though they are likely to struggle to get
traction at first. 
**Community Opinion** 
57% of those surveyed, or a total of 176 respondents, chose to answer
questions pertaining to this theme. We begin by reporting the questions asked and the breakdown of responses. 
**1. How relevant is this Theme for your own research?** 92% of respondents said it was somewhat relevant (23%), relevant (37%) or very relevant (32%). 
**2. Do you think neural approaches
alone are sufficient to achieve general
purpose AI agents that match or
surpass human intelligence in all
ways?** 16% of respondents said yes,
with the remainder saying no.. 
**3. What percentage of AI research should be devoted to combining
neural approaches with other approaches?** 94% said at least 25%, including those who said 25% (31% of respondents), 50% (35%) or greater than 50% (28%). One respondent answered 0%. 
**4. What percentage of AI research
should be devoted to purely nonneural approaches?** 86% said at least
25%, including those who said 25% (37% of respondents), 50% (38%) or greater than 50% (11%). 3% (6 respondents) answered 0%. 
**5. What paradigms beyond neural networks do you think deserve the most research attention currently?**
This open-ended question received 3 responses as follows.
* “We need to understand whether the brain is quantum.”
* “Classic approaches to AI that focus on high-level cognition, rely on
structured representations, take a systems-level approach, incorporate
ideas from psychology, and aim for theoretical insight rather than
winning competitions.”
* “interdisciplinary collaboration and consideration of ethical and
societal implications are extremely important.” 

As for all the survey categories, it is important to keep in mind that the
respondents to this theme are self selected (a little more than half of all
respondents). Of these respondents, most indicated resonance with the
main message of this theme, namely that it is important to invest to some
extent in non-neural research. On the other hand, it wouldn’t be surprising if many of those who chose not to answer these questions feel differently. 

1. Angelopoulos, Anastasios N. and Stephen Bates, “Conformal Prediction: A Gentle Introduction”, Foundations and Trends in Machine Learning: Vol. 16: No. 4, pp 494-591. http://dx.doi.
org/10.1561/2200000101 (2023)
2. Valmeekam, Karthik, et al. “On the planning abilities of large language models (a critical investigation with a proposed benchmark).” arXiv preprint arXiv:2302.06706 (2023).
3. Valmeekam, Karthik, Kaya Stechly, and Subbarao Kambhampati. “LLMs Still Can’t Plan; Can LRMs? A Preliminary Evaluation of OpenAI’s o1 on PlanBench.” arXiv preprint
arXiv:2409.13373 (2024).

Diversity of AI Research Approaches

Expanding AI research to include diverse perspectives and
expertise from outside the core AI research 
**Main Takeaways**
* Incorporating perspectives of social scientists, ethicists, and policymakers, to ensure responsible and ethical development and deployment of AI technologies
* AI is not just an engineering discipline but a societal force that reshapes governance, culture, economy, and ethics, requiring holistic approaches for responsible development and deployment.
* ‘Intelligent support’ and tools that facilitate seamless collaboration between AI researchers and experts from diverse domains, ensuring ethical, explainable, and application-specific AI solutions. 

**Context & History** 
‘Research beyond the AI research
community’ emphasizes the
importance of expanding AI research
to include diverse perspectives and
expertise from outside the core AI
research community. There are three
directions to this expansion: First, we
must include perspectives of a.o. social
scientists, ethicists, digital humanities,
critical data/ communication / media
studies, STS, and other disciplines,
and policymakers, to ensure
responsible and ethical development
and deployment of AI technologies.
Second, researchers and practitioners
in disciplines that increasingly rely
on AI (e.g., biology, law, business,
neuroscience, cognitive science) may
also influence how we think about AI
itself. Finally, as multidisciplinary efforts
increase we can provide ‘intelligent
support’ and tools for the interaction. 
The societal, ethical, legal, and cultural
challenges posed by AI highlight
the necessity of a multidisciplinary
approach to its development and
deployment [1,2]. Issues such as
biased decisions, privacy breaches,
governance, accountability, and
inclusivity demand solutions
that extend beyond the scope of
engineering. Addressing these
challenges requires the integration of
perspectives from humanities, social
sciences, and other fields to ensure
that AI systems are aligned with human
rights, societal values, and global equity. 
This broader approach to AI emphasizes
that its development and impact
cannot be compartmentalized as
solely technical advancements or
applications within specific fields, i.e.
AI is no longer purely an engineering
discipline. Instead, it requires a holistic
understanding of AI as a transformative
force that reshapes societal structures,
cultural norms, economic models,
and ethical frameworks. Such an
approach considers AI not merely as a
tool for individual disciplines but as an
integrated phenomenon that influences
and is influenced by diverse societal
factors. By involving diverse expertise,
including ethicists, legal experts,
social scientists, and policymakers, we
can develop governance frameworks
and accountability mechanisms to
address concerns like bias, inequality,
and unintended societal impacts.
Efforts to enhance inclusivity and
diversity in AI design can also mitigate
risks and maximize the benefits of AI
applications. 
Additionally, “intelligent support” for
these efforts refers to a combination
of tools, frameworks, and methods
that facilitate effective integration of
AI into diverse fields and collaboration
between AI researchers and experts in
other fields. 
**Current State & Trends** 
* Social scientists and ethicists
are increasingly involved in
developing guidelines on how AI
developers should handle personal
data, preventing inappropriate
surveillance, and ensuring the
responsible use of AI. Also
governance frameworks such as the
EU’s AI Act [3] are being actively
discussed in order to ensure that AI
development is aligned with human
rights, justice and societal needs.
* AI has become a key tool for
healthcare [4], law [5], business [6],
etc., which increasingly influences
AI research. For example, AI
applications in fields demand
highly accurate, explainable, and
interpretable AI systems due to the
importance of accountability and
transparency. This has spurred more
research in explainable AI (XAI) as
well as AI algorithms tailored for
certain applications, such as cancer
analysis [7].
* Intelligent support’ and tools for
the interaction between AI & other
disciplines: So far non-AI tools such
as GitHub, Kaggle, and research
repositories have been actively used
by the communities. We expect
that intelligent capabilities may be
able to promote more productive
interactions. 

**Research Challenges** 
In formulating future research, here is a
set of core pillars that we need to take
into account:
* **Societal Dynamics:** AI alters the
fabric of human interaction, labour
markets, and societal governance.
Understanding how automation,
algorithmic decision-making,
and AI-driven systems impact
democracy, social justice, and
equality is critical to harness its
potential without exacerbating
existing disparities.
* **Ethical Integration:** Ethics must be
embedded into AI from its inception.
This includes navigating dilemmas
around privacy, accountability, and
fairness. Ethical considerations
must go beyond technical checks to
involve diverse cultural,
philosophical, and societal inputs
to create systems that are globally
adaptable and contextually relevant.
* **Legal and Regulatory Frameworks:**
We need to address the need for regulation and governance, including
reevaluation of existing regulations
for intellectual property, liability,
and human rights. A collaborative,
interdisciplinary effort is essential
to craft governance models that
increase trust and safety, safeguard
public interests and contribute to
increased responsible innovation.
Cultural Adaptation and
Diversity: As AI technologies
become pervasive across cultures
and geographies, they must
adapt to diverse social norms,
languages, and traditions. Ensuring
cultural sensitivity in AI design
and implementation promotes
inclusivity and equitable access to
technological benefits.
* **Education and Public Awareness:**
The broad impact of AI requires a
rethinking of education systems to
prepare future generations for an AIinfused world. This includes fostering
a transdisciplinary understanding of
AI among engineers, social scientists,
policymakers, and the general public
to build informed, empowered
societies.
* **Environmental Sustainability:** The
energy demands of AI development
and deployment highlight its
environmental footprint. A broad
approach to AI considers how it can contribute to sustainability efforts,
from optimizing resource allocation
to advancing climate solutions, while
minimizing ecological harm. 

The inclusion of other disciplines is
more than developing AI technologies
to understanding and guiding their
interaction with society, but also to
ensure that their influence is equitable,
inclusive, and beneficial across all
domains of human activity. 
In supporting these efforts and
promoting interaction between AI and
other disciplines, future research topics
can also include the following: 
* **User friendly AI development
platforms:** These can enable people
from various disciplines to develop
AI solutions or train a deep learning
model without much programming
experience. The platform can also
guide ethical development of AI.
Intelligent user interfaces and
visualization tools can allow people
to understand AI model outputs and
gain insights.
* **Domain Specific AI tools:** AI systems
tailored to the particular needs of
a field, such as medical diagnosis,
cyber security, law, etc. We look
forward to intelligent capabilities
that understand the needs of the
field workers, and provide seamless support for domain-specific
workflows and reasoning processes,
which should be much more
powerful than current tools such as
domain-specific pretrained models
or knowledge-based systems
* **AI enabled Collaboration Tools:**
Intelligent environments that assist
AI researchers and experts in other
fields to work together. 

**Community Opinion** 
We began by asking the community which activities they considered most important. The top three priorities were: 1) Promoting interdisciplinary collaboration in AI research 2)
Developing AI solutions for specific application areas, such as healthcare, law, and business, and 3) Enhancing public understanding of AI’s impact. The next tier of priorities included:
4) Integrating AI education into noncomputer science disciplines, 5) Establishing accountability and liability standards, and 6) Addressing societal and cultural impacts of AI through diverse disciplinary perspectives and promoting inclusivity in AI design.
While activities such as optimizing resource allocation and developing governance models received slightly lower ratings, the differences were not significant. Overall, the community
expressed a strong interest in pursuing a broad range of initiatives. 
When asked about key areas of focus, an overwhelming 88% of respondents identified healthcare as the top priority. This was followed by climate (50%), education (45%), and biology
(43%). Other areas of interest included manufacturing (38%), business (19%), law (18%), and finance (15%). The respondents listed the following areas as additional candidates: agriculture,
transportation, brain diseases, physics, elder care, disaster response, media analytics, and cybersecurity. 
Regarding tools to support interdisciplinary collaboration, the
highest-rated option was user-friendly AI development platforms (65%), followed by AI-enabled collaboration tools (57%) and domain-specific AI tools (53%). Beyond technological support,
the community suggested the following actions for fostering collaboration: educational support, incentives to
encourage cross-disciplinary work, formation of interdisciplinary research groups, and sustainable funding for long-term projects. As multidisciplinary efforts become increasingly important
to AI research, the community seemed to emphasize the urgent need for stronger support from diverse perspectives. 
Approximately one-third of all participants responded to this theme, with the majority (around 90%) identifying AI as their primary field. Although input from other disciplines was limited, the survey suggests a strong interest within our community
in integrating diverse perspectives into research and enhanced support for interdisciplinary collaboration. 

1. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed, Bias and Fairness in Large Language
Models: A Survey. Computational Linguistics, 50(3):1097–1179, 2024.
2. Youngsik Yun and Jihie Kim. CIC: A framework for Cultually-aware Image Captioning, International Joint Conference on Artificial Intelligence, 2024.
3. The EU Artificial Intelligence Act, Regulation (EU) 2024/1689,
https://artificialintelligenceact.eu/
4. AI in health care, nature collection 22, 2024. https://www.nature.com/collections/dbfcjjigbi
5. D.W. Kite-Jackson2023 Artificial Intelligence (AI) TechReport, American Bar Association, 2023.
6. McKinsey & Company, The state of AI in early 2024: Gen AI adoption spikes and starts to generate value, 2024.
7. Joseph D. Janizek, Ayse B. Dincer, Safiye Celik, Hugh Chen, William Chen, Kamila Naxerova, and Su-In Lee. Uncovering expression signatures of synergistic drug responses via
ensembles of explainable machine-learning models. Nature Biomedical Engineering 7, 811–829, 2023. https://www.nature.com/articles/s41551-023-01034-0

Research Beyond the AI Research Community

State-of-the art AI is now largely driven by the private sector,
and universities struggle to compete: they need to find a role in
the new era of “big AI”. 
**Main Takeaways**
* The centre of gravity of AI research now lies behind the closed doors of big tech companies.
* Universities cannot compete with big tech companies with respect to the resources – data, compute, and salaries – that are being mobilised by the private sector.
* Universities struggle to retain AI faculty, and struggle to persuade AI graduates to remain in academia.
* The challenge is therefore now to find a role for academia (and publicly funded research) in the new era of “big AI”. 
 

**Context & History** 
While there has always been industrial interest in AI, for much of the history of the field, progress was driven largely by academia. The AI Turing award winners (Feigenbaum, McCarthy, Minsky, Newell, Pearl, Reddy, Simon) were university-based, and the key
concepts of both symbolic and neural AI were all developed within academia. The deep learning Turing award winners
(Bengio, Hinton, LeCun) all did their Laureate work in universities. But over the past decade, the centre of gravity in cutting edge AI has shifted decisively. In particular, primary work on generative AI is now being carried out largely within the private sector, and
a small number of big tech companies in particular. These companies have the data and compute resources available
to train large scale state of the art AI systems, and moreover are able to offer salaries that universities could not compete with. 
**Current State & Trends** 
These changes raise a number of issues for universities. 
First, the demand for AI research talent has led to an exodus of AI talent from universities to the private sector. In one
notably early case, Uber hired an entire research lab of roboticists from CMU [1]. Even the world’s leading universities are unable to offer packages to compete with the startling salaries offered by
big tech AI labs; nor can they muster the associated research resources that such companies routinely offer. The upshot has been a hollowing-out of many university AI groups, with
many faculty on seemingly permanent leave, or on part-time contracts that leave them largely disconnected from the conventional university business. Hiring AI faculty has been extremely challenging for more than a decade: PhD graduates prefer to move directly from their studies into the private
sector. 
At the same time the nature of AI research has changed dramatically. At the turn of the century the dominant paradigm with (for example) the AAAI community was conventionally scientific, drawing from mathematics (definition-lemma-theorem-proof). By contrast, the methodology of deep learning and its associated areas, which dominates at the time of writing, is
predominantly that of engineering. 
As a consequence the demand within
universities for AI courses in particular,
and computing courses more generally,
has skyrocketed, and this, coupled with
the issues we refer to above, has placed
intense pressure on departments,
whose staff struggle to cope with the
level of demand. Some ivy league
computing departments report that
while they have been given what
amounts to a blank cheque for hiring
new faculty in high demand areas,
they simply find it impossible to hire
enough high-quality faculty. In some
departments, this situation has reached
crisis levels, with faculty reporting stress
and other mental health issues as a
consequence. 
There is a need for all students
(regardless of their discipline) to
become literate in AI, the use of AI
tools is changing education and the
way subject are taught, and at the same
time students in AI must be educated
in a more multi-disciplinary context to
be aware of the ethical, legal, societal
and economic implications of AI. This
in turn places further demands on
universities, who need to rethink how
they teach and assess students when
those students have powerful general
purpose AI tools available. 
One irony of the present situation
is that a key role of universities
historically has been to provide a talent
pipeline to satiate the demands of the
technology world for technical talent.
But universities struggle to achieve this
simply because they don’t have the
capacity – because of the hollowingout of university departments by those same companies. 
At the same time, the nature of cutting
edge AI research means that universities
are simply unable to carry out work
that competes with big private sector AI
labs. A recent estimate from Meta put
the cost of building their latest GPTclass models at ~ $440 million. That is several orders of magnitude outside
the scope of all but a tiny number of
the world’s very richest universities;
and most nation states would struggle
to put assemble such a package for a
national level initiative: in 2023, the UK
contemplated a sovereign AI capability,
and one option considered was building
a “BritGPT”. The proposal foundered
at an early stage because of (i) cost, (ii)
risk, and (iii) being seen to compete
with the private sector in an area where
innovation (and development costs)
have been led by the private sector
[2]. Recent developments such as
that around the Chinese open-source
DeepSeek Large Language model may
offer an opportunity in this regard as
it could make AI available at lower
costs – although it presents challenges
of its own with respect to privacy and
security. 
The objectives of the private sector are different from those of universities.
The private sector is driven mainly by
profit, whereas universities aspire to
contribute to society through research
and education. These goals can be
in competition with each other. One
effect is that results from the private
sector are not always fully available
for inspection or evaluation. Also,
the high costs of the experiments
implies that results are not always
reproducible. Academics have a role to
play in providing independent advice
and interpretations of these results
and their consequences. The private
sector focuses more on the short term,
and universities and society more on a
longer term perspective.
These observation together with the
massive costs of generative AI research
has also motivated calls for large public-funded
initiatives, for instance, the plan supported by European Commission President Ursula von der Leyen for a CERN for AI. This vision is supported and was actually initiated by a large
number of AI researchers in Europe gathered in the CAIRNE network. It is modelled after the renowned CERN for
Nuclear Research in Geneva, and should serve as an alternative and attractive research environment to big tech companies, and should solve problems of public interest. Numerous countries
have also developed their national strategies and funding programs for AI. 
**Research Challenges** 
* How should universities respond to the new era of “Big AI”?
* What form of AI research can universities/public sector research
most usefully engage in – what should be the AI research agenda of universities going forward?
* How should universities respond to the challenges of hiring and retaining AI faculty and students?
* How can publicly funded universities best work with big tech AI
companies? 

**Community Opinion** 
Most respondents (approx. 75%)
agreed the universities have difficulty
recruiting in AI, and agreed that
universities have difficulty in engaging
in resource intensive AI research (80%).
Respondents felt that offering better
salaries and investing in better compute
resources were ways of attracting
people; offering joint appointments
and providing other benefits were also
seen as possibilities. There was less
agreement on whether universities
needed to refocus their research
priorities, although it seems that
respondents felt that focussing on
theoretical AI and multidisciplinary
AI were areas where universities
could be competitive. Public sector
funding for large scale compute
was seen as attractive (70%). There
was overwhelming agreement that
academia was very relevant to the
future of AI research. 
1. https://www.fastcompany.com/3046902/carnegie-mellon-in-a-crisis-after-uber-poached-40-of-its-researchers
2. https://lordslibrary.parliament.uk/large-language-models-and-generative-ai-house-of-lords-communications-and-digital-committee-report/

Role of Academia

Geopolitical Aspects & Implications of AI

AI and Scientific Integrity

AI evaluation is the process of assessing the performance, reliability, and safety of AI systems. 
**Main Takeaways** 
* AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods. 
* Current approaches to evaluation focus on benchmark-driven testing, e.g., of the quality of (generative) models, with insufficient attention paid to other critical factors such as usability, transparency, and adherence to ethical guidelines. 
* New insights and methods for evaluating AI systems are needed to provide the assurance for trustworthy, wide-scale deployments. 

**Context & History** 
The recent advances in AI have spurred tremendous innovation in potential applications for the technology. However, many organizations are hesitant to deploy AI systems due to risks that include reputational damage from generative AI hallucinations, leakage of proprietary data, and lack of assurance that legal and ethical guardrails will be enforced. 
Empirical methods have long played a role in AI research (e.g., [1]). Indeed, the research community has developed a robust body of metrics and methods for evaluating individual AI algorithms that has enabled the field to quantify performance and track progress. In contrast, less attention has been paid to evaluating AI systems and their deployment in real-world settings, including their usage by non-AI experts. 
AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods. The generality, complexity, and breadth of AI capabilities makes it impossible to test them exhaustively, requiring new thinking as to what constitutes sufficiency in testing. Run-time adaptivity and the evolution of learned models can change system behavior on the fly, introducing the need for continuous monitoring and validation. Many AI systems are designed to be used interactively, making collaborative usage and its impact on humans an important consideration. 
**Current State & Trends** 
Current practice for evaluating generative AI systems focuses on model-level testing relative to a growing body of benchmarks. Some benchmarks seek to measure general capabilities (e.g., GLUE [2], ARC-AGI [3], MMLU [4]) while others address particular types of reasoning and knowledge (e.g., MATH for mathematics [5], GPQA for logic [6], HumanEval for coding [7]). Benchmark-driven testing provides valuable insights into capabilities and shortcomings, as well as a principled means to evaluate progress over time. Benchmarks are used as proxies for AI capabilities but have an inherent contextualization that does not necessarily generalize well to new domains. Furthermore, benchmark-based testing is insufficient for ensuring successful deployment given the lack of attention to expected usage in real-world settings and aspects of human use. Benchmark-driven evaluation further raises issues related to overfitting and contamination of test data with training data. As embodied in Goodhart’s law, *“When a measure becomes a target, it ceases to be a good measure”*. 
Evaluating AI systems is inherently complex, especially if these systems are broadly applicable and capable of learning after deployment, requiring a balanced approach that is clear and transparent while avoiding overfitting to specific metrics at the expense of broader reliability, fairness, and real-world applicability. System-level evaluation, when done, explores representative use cases rather than seeking to be comprehensive. Redteaming serves as a complementary method through the use of adversarial interactions to identify misalignment with desired behavior models. 
Moving forward, AI evaluation needs to consider multiple dimensions of a system’s performance. Most evaluation efforts focus on capability, i.e., producing correct answers or behaviors in response to queries or tasking, for the scope of problems over which the system is expected to operate. Meeting capability requirements is essential for use; however, other aspects of performance must be considered. 
*Usability* is another critical dimension for evaluation. A principal factor of usability is transparency, meaning that mechanisms are provided that enable users to understand the basis for system actions and responses. Usability further requires directability, meaning that users can control and modify the behavior of the system to meet current and specialized needs (now often referred to as “alignment”). For AI systems being deployed to aid humans, evaluation must necessarily consider whether the technology ultimately improves combined humansystem performance. 
*Adherence* to legal requirements and ethical guidelines constitutes another important dimension for evaluation. Increasingly, geo-political entities are introducing legislation to restrict what and how AI systems will be allowed to operate within their jurisdictions, requiring validation that performance will stay within defined guardrails. Both government and commercial organizations have ethical and financial motivations to ensure that their use of AI is fair and unbiased. To support these goals, various trustworthy AI assessment frameworks have been developed to guide organizations in evaluating AI systems for fairness, transparency, robustness, and compliance with ethical standards. Notable frameworks include the EU Trustworthy AI Assessment Framework, the NIST AI Risk Management Framework, and the ISO/IEC 42001:2023 AI Management System Standard. 
AI systems introduce multiple operational issues related to their deployment. Privacy is a main concern: protecting personal or corporate information within a model from being leaked. AI systems have become attack surfaces themselves, with adversaries seeking to exfiltrate data or model weights, or to bias responses for purposes at odds with the model’s creators. Resource consumption and the cost for both training and deployment are additional considerations in evaluating overall performance of an AI system. 
These various factors must be weighed together, including fairness, robustness, interpretability, and compliance with evolving regulations. A comprehensive evaluation framework must balance these diverse considerations, ensuring AI systems are secure, efficient, and aligned with ethical and legal standards. 
**Research Challenges** 
There is a need for a science of evaluation for AI systems that will inject additional rigor into the evaluation process. This science will build on existing metrics and methodologies but incorporate new approaches that will increase confidence in our ability to deploy AI systems in missioncritical settings (e.g., [8] for evaluating Retrieval-Augmented Generation systems). Frameworks for auditing and reproducibility will be important to ensure the reliability and robustness of results [9]; as well, more attention should be paid to education within the field on proper empirical methodology. Below are key challenges for advancing our understanding of how to conduct effective evaluations for AI systems.
* Develop a better understanding how to monitor and assess AI systems that are deployed over extended periods of time, especially for those that evolve their behavior. 
* Develop frameworks for evaluating the safety of agentic AI systems that can take actions in the world. 
* Create methods to provide increased transparency into machine learning models. 
* Develop evaluation methodologies that directly address human engagement with AI capabilities (as was the case with the Turing test). 
* Understand the trade-offs between different dimensions of evaluation, such as whether increased transparency justifies higher costs, or adherence to guardrails outweighs potential impacts on privacy, or how other crossdimensional considerations might influence overall outcomes.
 
**Community Opinion** 
The responses to the community survey show that there is significant concern regarding the state of practice for evaluating AI systems. More specifically, 75% of the respondents either agreed or strongly agreed with the statement “The lack of rigor in evaluating AI systems is impeding AI research progress.” Only 8% of respondents disagreed or strongly disagreed, with 17% neither agreeing nor disagreeing. These results reinforce the need for the community to devote more attention to the question of evaluation, including creating new methods that align better with emerging AI approaches and capabilities. 
Given the responses to the first question, it is interesting that only 58% of respondents agreed or strongly agreed with the statement *“Organizations will be reluctant to deploy AI systems without more compelling evaluation methods.”* Approximately 17% disagreed or strongly disagreed with this statement while 25% neither agreed nor disagreed. If one assumes that the lack of rigor for AI research transfers to a lack of rigor for AI applications, then the responses to these two statements expose a concern that AI applications are being rushed into use without suitable assessments having been conducted to validate them. 
For the question *“What percentage of time do you spend on evaluation compared to other aspects of your work on AI?”* the results show 90% of respondents spend more than 10% of their time on evaluation and 30% spend more than 30% of their time. This clearly indicates that respondents take evaluation seriously and devote significant effort towards it. While the prioritization of evaluation is commendable, the results would also seem to indicate that evaluation is a significant burden, raising the question of what measures could be taken to reduce the effort that it requires. Potential actions might include promoting an increased focus on establishing best practices and guidelines for evaluation practices, increased sharing of datasets, and furthering the current trend of community-developed benchmarks. 
The most widely selected response to the question* “Which of the following presents the biggest challenge to evaluating AI systems?”* was a lack of suitable evaluation methodologies (40%), followed by the black-box nature of systems (26%), and the cost/ time required to conduct evaluations (18%). These results underscore the need for the community to evolve approaches to evaluation that align better with current techniques and broader deployment settings. 
1. Cohen, P.R. (1995). Empirical Methods for Artificial Intelligence. MIT Press. 2. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. BlackboxNLP@ EMNLP. 3. Chollet, F. (2019). On the Measure of Intelligence. ArXiv, abs/1911.01547. 4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.X., and Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. ArXiv, abs/2009.03300. 5. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D.X., and Steinhardt, J. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. ArXiv, abs/2103.03874. 6. Rein, D., Hou, B.L., Stickland, A.C., Petty, J., Pang, R., Dirani, J., Michael, J., and Bowman, S.R. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. ArXiv, abs/2311.12022. 7. Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. ArXiv abs/2107.03374 8. Shahul, E., James, J., Anke, L.E., and Schockaert, S. (2023). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Conference of the European Chapter of the Association for Computational Linguistics. 9. Gundersen, O.E., Helmert, M., and Hoos, H. (2024). Improving Reproducibility in AI Research: Four Mechanisms Adopted by JAIR. J. Artif. Intell. Res. 81, 1019-1041.

Downloads

Next from The Future of AI Research - Webinar Series

AI agents

Similar lecture

AI Evaluation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The Future of AI Research - Webinar Series

AI agents

Similar lecture

AI Evaluation

Downloads