superintelligent agents
karim jebari and joakim lundborg
outline
- why does it matter?
- what is a machine agent?
- the dynamics of agency
- conclusion
value alignment
- superintelligence
- general agency
two scenarios
- spontaneous emergence
- accidental emergence
agency and desire
The generality of an agent is related to the productivity of its desires
productivity
a desire is productive to the extent that it can direct behaviour in different situations.
this is often done by generating new desires relevant to the context
example
Paperclip ai has a very productive desire. It may seem narrow, but it can direct behaviour in a wide variety of of contexts.
Alphazero has very unproductive desires.
can specialized AI acquire productive desires?
a desire can only be acquired from a set of pre-existing desires, an AI with a set of desires constrained to a specific domain cannot acquire desires relevant to other domains.
the humean model
belief
desire
directions of fit
- belief: mind to world
- desire: world to mind
learning requires reinforcement
- the world can reinforce our beliefs, but not our desires
- desires can only be reinforced "from within"
so this is wrong
AI cannot become a general agent sui generis
objections
the second scenario
self-preservation?
natural selection
pain
conclusions
creating a general AI agent requires a concerted effort
Thank you!
Karim Jebari
jebarikarim@gmail.com
politiskfilosofi.com
twitter.com/karimjebari
