Neural Actors:
The intersection between
machine learning, hardware,
and the actor model

@Smerity

My history is a mix of

linguistics

and

computing

Leveraging more compute

Actors

ML

Will Wright's "Dynamics for Designers"

Rich interactions for systems we don't yet know / can't explicitly specify we're developing

An actor can:

- Send a finite number of messages to other actors

- Actors have a mailbox with an address and bounded / unbounded capacity

- Create a finite number of new actors

The last point minimizes the need for orchestration systems

The actor model

The actor model helps solve:

- Concurrency (no shared state, many cores)

- Scalability (spawn new actors as needed, potentially on nodes across the network)

- Reliability (actors supervising other actors)

- Flexibility: "Actor creation plus addresses in messages means variable topology"

The actor model

Erlang / Elixir:

- Actors as a language primitive
- Erlang/OTP used for reliable telecoms

Rust's "fearless concurrency":

- Mutable with single owner
- Immutable with many references

The actor model in Software 1.0

Actor model enabling Amazon's two pizza rule

WhatsApp: "35 engineers and reached more than 450 million users" (pre Facebook acquisition)

Discord: "scaling to over a 100 million messages per day with only 4 backend engineers in 2017 and serving 250+ million users with less than 50 engineers in 2020"

Team and software scaling

Actors and ML

ML frameworks act as message passing++
(fwd + bwd are sync / async msgs)

Actors are explicit operations learning and performing implicit tasks via obj functions

ML components can be seen as actors but:
- High parallelism, minimal concurrency (SPMD)
- Inability to spawn (except limited by the above)

The limitations of hardware

SPMD means "one (hammer) program"
The result? All problems made equivalent nails

Multi-tenancy would provide different primitives
(at least CPUs are ~good at time sharing)

At present any "spawning" is manually specified
Remember: "Actor creation plus addresses in messages means variable topology"

What does MPMD look like atm?

- NVIDIA: High end cards at best 7 MPMD (MIG)
(max theoretical is 108 as 108 SMs)

...

The best you can do is run many nodes with many cards and send messages about

This gives you the horrors of both worlds: neural networks and container orchestration!

Mixture of Experts (MoE)

Neural actors

A neural actor can:

- Send a finite number of messages to other actors
(Explicit addr or implicit addr via attention)

- Actors have a mailbox with an address and bounded / unbounded capacity

- Create a finite number of new neural actors

Neural actor possibilities

Scale up/down network and compute
- Proxy actors for messaging (filter out at source, predict missing packets, ...)
- LM actors spawned between components for shared compressed language / comms
- TEMPEST actors: "ephemeral arbitrator between AIs w/o knowledge exposed"
- Spare capacity for expansion / distillation
- Treat msgs over network like RNN BPTT

Example: Expert Choice MoE

Ancient history: n-grams

Past decades: search engine's inverted index
(past: "Actor" appears on pages A, B, C, ...)

Recent:
word2vec("Actor") => 1024 dim f32 vector
"Actor" + context => 1024 dim vector

The word connecting to its own meaning

Know a word by the company it keeps

Data actors

An actor expanding / better understanding a specific piece of data (implicit program)

- Naive: inverted index (i.e. search)
- Implicit: language models / embeddings
- Explicit: data actors continuously shifting about an actor ecosystem

"Actor" node has high entropy .: mitosis spawns: "Actor (programming)", "Actor (arts)"

Traveling Linguist Problem

Data actors shifting about an actor ecosystem

=>

A data ecosystem groups, sorts, and removes redundancies within itself such that you have the minimal surprise learning a language

Will Wright's "Dynamics for Designers"

Rich interactions for systems we don't yet know / can't explicitly specify we're developing