Russian Federation

&quot;While it is still unclear if agents with Artificial General
Intelligence (AGI) could ever be built, we can already use
mathematical models to investigate potential safety systems for these
agents. We present work on an AGI safety layer that creates a special
dedicated input terminal to support the iterative improvement of an
AGI agent&#39;s utility function. The humans who switched on the agent
can use this terminal to close any loopholes that are discovered in
the utility function&#39;s encoding of agent goals and constraints, to
direct the agent towards new goals, or to force the agent to switch
itself off.

An AGI agent may develop the emergent incentive to manipulate the
above utility function improvement process, for example by deceiving,
restraining, or even attacking the humans involved. The safety layer
will partially, and sometimes fully, suppress this dangerous
incentive.

This paper generalizes earlier work on AGI emergency stop buttons. We
aim to make the mathematical methods used to construct the layer more
accessible, by applying them to an MDP model. We discuss two provable
properties of the safety layer, identify still-open issues, and
present ongoing work to map the layer to a Causal Influence Diagram
(CID).&quot;

AGI-20

Towards AGI Agent Safety by Iteratively Improving the Utility Function

corrigibility

provable safety

safety layer

agi safety

The AGI conference series is the only major conference series devoted wholly and specifically to the creation of AI systems possessing general intelligence at the human level and ultimately beyond. By gathering together active researchers in the field, for presentation of results and discussion of ideas, we accelerate our progress toward our common goal.

Underline hosted the two main days of AGI 2020 on Wednesday June 24th and Thursday June 25th. Both days were **live-streamed** and you can read the full program bellow. All presentations are available for a **asynchronous viewing**. All presenters are grouped into panels. You can also hear guest talks by **Ben Goertzel** (SingularityNET), **Valery Tarasov** (Semenov Institute of Chemical Physics), **Tatiana Shavrina** (Sberbank) and **Joscha Bach** (MIT Media Lab). 

The AGI conference series is the only major conference series devoted wholly and specifically to the creation of AI systems possessing general intelligence at the human level and ultimately beyond. By gathering together active researchers in the field, for presentation of results and discussion of ideas, we accelerate our progress toward our common goal.

"While it is still unclear if agents with Artificial General
Intelligence (AGI) could ever be built, we can already use
mathematical models to investigate potential safety systems for these
agents. We present work on an AGI safety layer that creates a special
dedicated input terminal to support the iterative improvement of an
AGI agent's utility function. The humans who switched on the agent
can use this terminal to close any loopholes that are discovered in
the utility function's encoding of agent goals and constraints, to
direct the agent towards new goals, or to force the agent to switch
itself off.

An AGI agent may develop the emergent incentive to manipulate the
above utility function improvement process, for example by deceiving,
restraining, or even attacking the humans involved. The safety layer
will partially, and sometimes fully, suppress this dangerous
incentive.

This paper generalizes earlier work on AGI emergency stop buttons. We
aim to make the mathematical methods used to construct the layer more
accessible, by applying them to an MDP model. We discuss two provable
properties of the safety layer, identify still-open issues, and
present ongoing work to map the layer to a Causal Influence Diagram
(CID)."

technical paper

It is often useful for one agent to predict what another agent will believe after receiving new information. In fact, in order to appear intelligent in situations involving multiple interacting agents, we fundamentally need to be able to predict how changes in the world will affect the beliefs of others. This process involves two distinct processes. First, we need to devise a model that captures the way that beliefs change in response to new information. Second, we need to observe the behaviour of individual agents to determine their specific beliefs. In the AI literature, these problems have been addressed by distinct communities. In this paper, we bring these two communities together by demonstrating how an agent can learn a model of belief change from observed behaviour. We argue that this process is essential for natural interaction in an AGI setting, but it has not been addressed to date in a unified manner.

Learning to Model Another Agent's Beliefs: A Preliminary Approach

The paper proposes the architecture of dynamically changing hierarchical memory based on compartmental spiking neuron model. The aim of the study is to create biologically-inspired memory models suitable for implementing the processes of features memorizing and high-level concepts. The presented architecture allows us to describe the bidirectional hierarchical structure of associative concepts related both in terms of generality and in terms of part-whole, with the ability to restore information both in the direction of generalization and in the direction of decomposition of the general concept into its component parts. A feature of the implementation is the use of a compartmental neuron model, which allows the use of a neuron to memorize objects by adding new sections of the dendritic tree. This opens the possibility of creating neural structures that are adaptive to significant changes in the environment.

The hierarchical memory based on compartmental spiking neuron model

Artificial intelligence becomes an integral part of human life. At the
same time, modern widely used approaches, which work successfully due to the availability of enormous computing power, based on ideas about the work of the brain, suggested more than half a century ago. The proposed model describes the general principles of information processing by the human brain, taking into account the latest achievements. The neuroscientific grounding of this model and
its applicability in the creation of AGI or Strong AI are discussed in the article.
In this model, the cortical minicolumn is the primary computing processor that works with the semantic description of information. The minicolumn transforms incoming information into its interpretation to a specific context. In this way, a parallel verification of hypotheses of information interpretations is provided when comparing them with information in the memory of each minicolumn of the cortical zone, and, at the same time, determining a significant context is the information transformation rule. The meaning of information is defined as an interpretation that is close to the information available in the memory of a minicolumn. The behavior is a result of modeling of possible situations. Using this approach will allow creating a strong AI or AGI.

The meaning of things as a concept of strong AI architecture

This paper introduces the open-source software project, Brain Simula-tor II, simplifying experimentation into various facets of AGI. The software seamlessly marries spiking neural networks with symbolic AI algorithms. It sup-ports a large array of simple neurons (of various models) and groups of neurons collected into “Modules”, backed by custom software. 3D and 2D simulators al-low a virtual entity to move about, have binocular vision and touch, and merge this information with spoken input. Information is captured in a Universal Knowledge Store module which represents information in links between nodes. Continuing development will enhance these capabilities.

New Brain Simulator II Open-Source Software

This paper compares the various conceptions of “real-time” in the context of AI, as different ways of taking the processing time into consideration when problems are solved. An architecture of real-time reasoning and learning is introduced, which is one aspect of the AGI system NARS. The basic idea is to form problem-solving processes flexibly and dynamically at run time by using inference rules as building blocks and incrementally self-organizing the system’s beliefs and skills, under the restriction of time requirements of the tasks. NARS is designed under the Assumption of Insufficient Knowledge and Resources, which leads to an inherent ability to deal with varying situations in a timely manner.

An Architecture for Real-time Reasoning and Learning

Even with the relative ascendancy of sub-symbolic approaches to AI, the symbol grounding problem presents an ongoing challenge to work in artificial general intelligence. Prevailing ontology design practices typically presuppose the transparency of the relation between semantics and syntax by transcendently stipulating it extrinsic to the system, rather than providing a platform for the internal development of this relation through agents’ interactions endogenous to the system. Drawing on theoretical resources from ecological psychology, dynamical systems theory, and interactive computation, this work suggests an inversion of the symbol grounding problem in order to analyze how the symbolic regime can emerge out of causally embedded dynamical interactions within a system of autonomous intelligent agents. Under this view, syntactic-symbols come to be stabilized from other signs as constraints harnessing the dynamics of agents’ inter-actions, where the functional effects generated by such constrained dynamics give rise to an internal characterization of semantics broadly aligned with Brandom’s semantic pragmatism. Finally, ludics—a protological framework based on interactive computation—provides a formal model to concretely describe this continuity between syntax and semantics arising within and through the regulative dynamics of interactions in a multi-agent system. Accordingly, this bottom-up approach to grounding the symbolic order in dynamics could provide the conditions for artificial agents to engage in autonomous language design, thus equipping themselves with a powerful cognitive technology for intersubjective coordination and (re)structuring ontologies within a community of agents.

The Dynamics of Growing Symbols: A Ludics Approach to Language Design by Autonomous Agents

There are different difficulties in defining a fundamental concept; it often happens that some conditions are too strong or just surplus, and others are too weak or just lacking. There is no clearly agreed conception of intelligence, let alone artificial intelligence and artificial general intelligence. Still it can be significant and useful to (attempt to) elucidate the defining or possible characteristics of a fundamental concept. In the present paper we discuss the conditions of artificial general intelligence, some of which may be too strong and others of which may be too weak. Among other things, we focus upon logic, autonomy, resilience, integrity, morality, emotion, embodiment, and embeddedness, and articulate the nature of them from different conceptual points of view. And we finally discuss how to test artificial general intelligence, proposing a new kind of Turing-type tests based upon the intelligence-for-survival view. Overall, we believe that explicating the nature of artificial general intelligence arguably contributes to a deeper understanding of intelligence per se.

The Conditions of Artificial General Intelligence

Agents are designed to perform specific tasks. The agent developers define the agent’s environment, the task states, the possible actions to navigate the different states, and the sensors and effectors necessary for it to perform its task. Once trained and deployed, the agent is monitored to ensure that it performs as designed. During operations, some changes that were not foreseen in the task design might negatively impact the agent performance. In this case, the agent operator would capture the performance drop, identify possible causes, and work with the agent developer to update the agent design. This model works well in centralized environments. However, agents are increasingly deployed in decentralized, dynamic environments, where changes are not centrally coordinated. In this case, updating agent task design to accommodate unforeseen changes might require a considerable effort from the agent operators. The paper suggests an approach to enable agents to anticipate and identify deviations in their performance on their own, thus improving the process of adapting to changes. The approach introduces an additional machine learning-based component—we call information digital twin (IDT)—dedicated to predicting task changes. That is, an agent would then have two components: the original component, which focuses on finding the best actions to achieve the agent task, and the IDT, dedicated to detecting changes impacting the agent task. Considering general artificial intelligence agents—where an agent might manage different tasks in various domains—the proposed IDT might be a component that enables AGI agents to ensure their performance against changes.

Information Digital Twin—Enabling Agents to Anticipate Changes in their Tasks

I give a brief overview of the theory I have depicted in my paper called "Cognitive Machinery and Behaviours".

Cognitive Machinery and Behaviours

The ability to explain decisions made by AI systems is highly sought after, especially in domains where human lives are at stake such as medicine or autonomous vehicles. While it is often possible to approximate the input-output relations of deep neural networks with a few human-understandable rules, the discovery of the double descent phenomena suggests that such approximations do not accurately capture the mechanism by which deep neural networks work. Double descent indicates that deep neural networks typically operate by smoothly interpolating between data points rather than by extracting a few high level rules. As a result, neural networks trained on complex real world data are inherently hard to interpret and prone to failure if asked to extrapolate. To show how we might be able to trust AI despite these problems we introduce the concept of self-explaining AI. Self-explaining AIs are capable of providing a human-understandable explanation of each decision along with confidence levels for both the decision and explanation. Some difficulties with this approach along with possible solutions are sketched. Finally, we argue it is important that deep learning based systems include a "warning light" based on techniques from applicability domain analysis to warn the user if a model is asked to extrapolate outside its training distribution.

Downloads

Next from AGI-20

Learning to Model Another Agent's Beliefs: A Preliminary Approach

Similar lecture

Experience-specific AGI Paradigms

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AGI-20

Learning to Model Another Agent's Beliefs: A Preliminary Approach

Similar lecture

Experience-specific AGI Paradigms

Downloads