Elisa Neural
Computer Agent

An experimental agent that learns through a growing library of small experts.

An illustrated desktop with a cursor moving a file manager window

Core Idea

Most frontier AI models are giant monolithic models. Models consisting of trillions of parameters are not uncommon. The unfortunate consequence of this is that it requires an enormous amount of memory, data, and compute both for training and inference. This makes it borderline impossible for anything but large corporations and governments to train, control and run.

The Elisa Neural Computer Agent attempts to solve this issue by using an architecture that consists of 2 parts. A kernel which does the housekeeping, and experts which are small neural or neuro-symbolic entities that can be dynamically loaded and evicted on demand.

You only pay for what you use

Regardless of the total amount of experts the system consists of, only the models that are needed at a given time are loaded into memory. This means simple operations require very little resources, while more complex ones needs more.

The additional benefit is that the entire system need not be trained at once. The modular nature makes training experts in isolation possible on consumer hardware. People all over the world can contribute without the finanical backing of investors.

What we want Elisa Neural Computer Agent to learn

Cognitive world model
Most AI models aspire to create as rich world model as possible in order to be fully grounded in the real world. This means training on gigantic datasets to store as much knowledge as possible. A more sensible approach would be to fully tackle the building blocks of cognition first, which encompasses things like working memory, planning, cause and effect, understanding of time, space, shapes, sound, motion, and distance. If an agent develops a rich abstract world model, it should later be able to adapt much faster to a physical world model. Understanding the physical world to adapt to an abstract world is backwards.
Control a computer in real time
In order to be a truly effective user of a computer the agent needs to be able to work with continuous streams of sensory data like vision and audio with minimal latency and perform split second decision making. It should use adaptive thinking in order to decide when to react quickly and when to plan its next moves.
Learn when to change
A single missed click might be noise. Repeated misses after an interface update might justify learning a new response. The aim is continual learning with self-regulating plasticity, where evidence guides how readily the agent changes a skill it already uses.
Keep knowledge and retrieve it later
Working memory holds recent observations and actions. Long-term memory stores learned experts and the shared representations they use, so a skill can be fetched from disk when a later task needs it. The project is testing how a learner uses recent experience as feedback.

How capability should grow

Let people train individual experts
Someone with useful examples and a home computer should be able to train one specialist and share it worldwide. Each expert declares its inputs and outputs so the kernel can connect independently developed skills.
Leave room for richer perception
A skill learned from a low-resolution screen should remain useful when a sharper image or audio becomes available. The design should allow new ways to interpret input beyond the resolution and parameter count of any one expert.
Bigger computer, better performance
A faster CPU or GPU should let the agent evaluate more candidate actions within the same response time. More memory should let it work with more experts at once. A model need not be re-trained to take advantage of better hardware.
Run locally on your own hardware
Hardware is getting faster and cheaper, this makes running capable AI system locally with low latency more feasible every year that passes by. But just as important as expanding capability if not more important, is the ability to get more from less. If the state of the art is getting more powerful yet at the same time slower at inference, this is not progress. Real progress should be more capability in LESS time than the last generation.
Design for years of extension
Its plug and play nature means that it can be as relevant five years from now as it is today. Experts can be added, removed or re-trained, and continuous improvements to the kernel will increase its usefulness.

Make skills easy to use, teach and share

The goal is a package manager where people can find an expert or package of experts, install it, train it on examples and distribute an update. Each step should be approachable for someone who knows the task well but has never trained a machine-learning model.

Permissive Open Source licence

The licensing goal is to make Elisa Neural Computer Agent easy to use, study, modify and share.

Work in progress

The kernel already plans routes, loads and evicts experts, and checks new experts against fresh evidence. Learning from screen rewards is under test. Reliable real-time desktop control and the contributor experience described here are project goals.

But still the project is very much a work in progress, and is currently to be considered an experimental research project. Videos and updates to capabilities will be demonstrated continuously.