Temporal Integration in the Mesolimbic Pathway during Pavlovian Conditioning

Daniel Fürth, DMC lab meeting 8 december 2014

Content

The Clinical Problem
Historical background
Walkthrough of the project
Things to be done

Main Argument

The brain does not compute running averages of past events (like reinforcement learning models).
The brain has a rich temporal representation of events and their relation in time (a temporal map) just as rich as the spatial map.
The organism gathers information about its environment and then makes decisions on grounds of this knowledge when it wants to.
This view is essential to understand both aversive and appetitive learning.

Why I don't care much about computational neuroscience

When a man throws a ball high in the air and catches it again, he behaves as if he had solved a set of differential equations in predicting the trajectory of the ball... At some subconscious level, something functionally equivalent to the mathematical calculation is going on.

Richard Dawkins, Selfish Gene

m\frac{d^{2}x}{dt^{2}} + cS\left(\frac{dx}{dt}\right)^{2} = 0, \quad m\frac{d^{2}y}{dt^{2}} + cS\left(\frac{dy}{dt}\right)^{2} = -mg,

m \frac{​ d ​ 2 ​ ​ x ​ ​}{​ d t ​ 2 ​ ​ ​} + c S (\frac{​ d x ​ ​}{​ d t ​}) ​ 2 ​ ​ = 0, m \frac{​ d ​ 2 ​ ​ y ​ ​}{​ d t ​ 2 ​ ​ ​} + c S (\frac{​ d y ​ ​}{​ d t ​}) ​ 2 ​ ​ = - m g,

\frac{d^{2}h_x}{dt^2} + k\left( \frac{dh_x}{dt} \right)^2 = 0, \quad h_x(t=0) = x_0 , \quad v_x = \left[\frac{dh_x}{dt}\right]_{t=0} = v_0cos(\alpha)

\frac{​ d ​ 2 ​ ​ h ​ x ​ ​ ​ ​}{​ d t ​ 2 ​ ​ ​} + k (\frac{​ d h ​ x ​ ​ ​ ​}{​ d t ​}) ​ 2 ​ ​ = 0, h ​ x ​ ​ (t = 0) = x ​ 0 ​ ​, v ​ x ​ ​ = [\frac{​ d h ​ x ​ ​ ​ ​}{​ d t ​}] ​ t = 0 ​ ​ = v ​ 0 ​ ​ c o s (α)

\frac{d^{2}h_y}{dt^2} + k\left( \frac{dh_y}{dt} \right)^2 = -g, \quad h_y(t=0) = x_0 , \quad v_y = \left[\frac{dh_y}{dt}\right]_{t=0} = v_0sin(\alpha)

\frac{​ d ​ 2 ​ ​ h ​ y ​ ​ ​ ​}{​ d t ​ 2 ​ ​ ​} + k (\frac{​ d h ​ y ​ ​ ​ ​}{​ d t ​}) ​ 2 ​ ​ = - g, h ​ y ​ ​ (t = 0) = x ​ 0 ​ ​, v ​ y ​ ​ = [\frac{​ d h ​ y ​ ​ ​ ​}{​ d t ​}] ​ t = 0 ​ ​ = v ​ 0 ​ ​ s i n (α)

Gaze heuristic (McLeod et al. 2003, Nature)

Why I don't care much about computational neuroscience

linear optical trajectory (LOT) model (Shaffer et al. 2008)

Why I don't care much about computational neuroscience

To study the brain properly you should begin, like David Marr, by asking what tasks is it performing. Not assuming what it is doing.

The Clinical Problem

Treatment refusal (Issakidis & Andrews, 2004)

30%

Exposure based therapy.

Treatment attrition (Haby et al., 2006)

15-30%

Treatment non-responders (Loerencic et al. in submission)

40-50%

Return of fear (Craske & Mystkowski, 2006)

19-62%

11-29% of the intent-to-treat population will experience a substantial clinical effect

If return of fear solved: 25-36% about 400'000 individuals in Sweden.

If non-responders solved: 23-53% about 670'000 individuals.

Momentary prevalence of anxiety disorders: 12-17%.

The Clinical Problem

The Economical Problem

How often should you meet your therapist?
How many sessions should you schedule?

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

sufficient and necessary conditions for learning

Similarity
Contiguity
Contrast
Frequency

Laws of recall:

Did Aristotle actually claim this?
De Memoria Et Reminiscentia (Ross & Aristotle, 1906, p. 111)

Historical background

sufficient and necessary conditions for learning

Similarity
Contiguity
Contrast
Frequency

Laws of recall:

Did Aristotle actually claim this?
De Memoria Et Reminiscentia (Ross & Aristotle, 1906, p. 111)

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

learning as an autocatalytic set

Russian physicist Schukarew (1907) inspired by the law of physical chemistry known as the autocatalytic principle.

Some reactions accelerated by external substance, a catalyst.

Other autocatalyst - the products of the reaction itself accelerate the process.

Amount of catalyst currently available.
The amount of chemical as yet unreacted.

Autocatalytic Principle:

Rate of reaction is proportional to:

V_x(t)

V ​ x ​ ​ (t)

[\lambda_x - V_x(t)]

[λ ​ x ​ ​ - V ​ x ​ ​ (t)]

Historical background

learning as an autocatalytic set

\text{Rate at which learning occurs} = (\text{Amount of learning that has already occured}) \times (\text{Amount of learning that has yet to occur}),

R a t e a t w h i c h l e a r n i n g o c c u r s = (A m o u n t o f l e a r n i n g t h a t h a s a l r e a d y o c c u r e d) \times (A m o u n t o f l e a r n i n g t h a t h a s y e t t o o c c u r),

In words:

Expressed as a first-order differential equation:

\frac{\text{d}V}{\text{d}t} = \alpha \cdot V(\lambda - V),

\frac{​ d V ​ ​}{​ d t ​} = α \cdot V (λ - V),

Where:

\alpha \text{ is a free learning rate parameter on the unit interval. } \alpha \in [0,1],

α i s a f r e e l e a r n i n g r a t e p a r a m e t e r o n t h e u n i t i n t e r v a l . α \in [0, 1],

Historical background

The shape of the learning curve

V_{xit} = \frac{\lambda_x}{(1 + \beta_x \text{exp}\{-\alpha_i \times TIME_t\})},

V ​ x i t ​ ​ = \frac{​ λ ​ x ​ ​ ​ ​}{​ ( 1 + β ​ x ​ ​ e x p { - α ​ i ​ ​ \times T I M E ​ t ​ ​ } ) ​},

\Delta V_{xi}(t) = \beta_x \alpha_i \cdot [\lambda_x - V_{xi}(t-1)]

Δ V ​ x i ​ ​ (t) = β ​ x ​ ​ α ​ i ​ ​ \cdot [λ ​ x ​ ​ - V ​ x i ​ ​ (t - 1)]

Linear operator:

Logistic learning curve:

Associative value / Response strength
Salience of stimuli
Learning rate for the individual.
Prediction-error.

i\text{th}

i t h

x

V_{xi}(t)

V ​ x i ​ ​ (t)

\beta_x

β ​ x ​ ​

\alpha_x

α ​ x ​ ​

[\lambda_x - V_{xi}(t - 1)]

[λ ​ x ​ ​ - V ​ x i ​ ​ (t - 1)]

Historical background

learning as an autocatalytic set

Why autocatalytic processes?

Historical background

learning as an autocatalytic set

Why autocatalytic processes?

At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.

Historical background

learning as an autocatalytic set

Why autocatalytic processes?

At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.
This theory is ofcourse altogether abandonned:
- Watson & Crick (1953), discovery of double helix
  - Central Dogma of Mol. Biology.

Historical background

learning as an autocatalytic set

Why autocatalytic processes?

At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.
This theory is ofcourse altogether abandonned:
- Watson & Crick (1953), discovery of double helix
  - Central Dogma of Mol. Biology.

Assumption: proteins as the basis of memory (synaptic plasticity).
Concentration = Synaptic weight = Strength of memory trace

No role for information.
Compare autocatalysm to DNA and translation.

Historical background

engrams and memory traces

Engram (Karl Lashley, Richard Thompson etc.)

David Hasley (1746)

Forgetting (see Bjork, 2003)

Trace decay (Thorndike, 1914)
Consolidation (Müller & Pilzecker, 1900)
Interference (Kahana, Brown, Lewandowsky)

Edward Throndike (1914) Law of Disuse

Consolidation: cannot explain proactive interference (old information that hinders retrieval of new information).

Decay theory: Completely destroyed by John McGeoch (1932). Mere passage of time doesn't lead to forgetting, what is done during this passage of time matters and can explain most.

Historical background

engrams and memory traces

Engram (Karl Lashley, Richard Thompson etc.)

David Hasley (1746)

Spike-Time-Dependent Plasticity (STDP) Markram et al. 1997

Edward Throndike (1914) Law of Disuse

RLT Credit assignment problem

Forgetting (see Bjork, 2003)

Trace decay (Thorndike, 1914)
Consolidation (Müller & Pilzecker, 1900)
Interference (Kahana, Brown, Lewandowsky)

Historical background

memory traces and prediction errors

The Backward View of Temporal Difference (TD)-reinforcement learning

Midbrain dopaminergic nuclei as a finite state machine.

Historical background

engrams and memory traces

Crick, (1997) TRSB

Crick, (1984) Nature

Of course memories are not stored at synapses. But I think it is useful to contemplate the possibility that they are not stored anywhere else in the brain either.
The whole issue of where or, more important, how memories are stored in the brain may turn out to be an incorrect formulation of the problem, despite its seductively enticing allure. And the source of such an incorrect formulation may lie in the single-minded preoccupation with the storage, or the engram, and sometimes even identification of storage with memory.

This preoccupation with the physical changes that follow from an experience that can be remembered seems to be accompanied by a rather conspicuous neglect of retrieval processes. - Endel Tulving

Historical background

engrams and memory traces

Historical background

engrams and memory traces

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

Tolman and Latent Learning

T-alley Maze

Tolman & Honzik, 1930

Phase I

Test

Tolman, Ritchie, and Kalish, 1946

Result

Historical background

Tolman's Cognitive Maps

Historical background

milleniums of learning theory

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

Bush & Mosteller, 1951

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

1960-70 contiguity hits the fan

Robert Rescorla's Truly random control.

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

1960-70 contiguity hits the fan

Wagner's Relative Validity.

Experimental group:

10 trials of Tone + Light with food.

10 trials of Click + Light without food

Control group:

5 trials of Tone + Light with food.

5 trials of Click + Light without food

5 trials of Tone + Light without food.

5 trials of Click + Light with food

Wagner, A.R., Logan, F.A., Haberlandt, K. & Price, T. (1968)

Historical background

1960-70 contiguity hits the fan

Wagner's Relative Validity.

Experimental group:

10 trials of Tone + Light with food.

10 trials of Click + Light without food

Control group:

5 trials of Tone + Light with food.

5 trials of Click + Light without food

5 trials of Tone + Light without food.

5 trials of Click + Light with food

Only conditioning to the light in the control group.

Wagner, A.R., Logan, F.A., Haberlandt, K. & Price, T. (1968)

The animal is attending more to a stimulus that constantly predicts the outcome and attending less to a poor predictor.

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

1960-70 contiguity hits the fan

Patricia Janak (2013) Nature Neuroscience

Kamin's Blocking Effect

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

Rescorla-Wagner model 1972

\Sigma V = V_A + V_B

Σ V = V ​ A ​ ​ + V ​ B ​ ​

Backward blocking

\Delta V_{xi}(t) = \beta_x \alpha_i \cdot [\lambda_x - V_{xi}(t-1)]

Δ V ​ x i ​ ​ (t) = β ​ x ​ ​ α ​ i ​ ​ \cdot [λ ​ x ​ ​ - V ​ x i ​ ​ (t - 1)]

Associative value / Response strength
Salience of stimuli
Learning rate for the individual.
Prediction-error.

i\text{th}

i t h

x

V_{xi}(t)

V ​ x i ​ ​ (t)

\beta_x

β ​ x ​ ​

\alpha_x

α ​ x ​ ​

[\lambda_x - V_{xi}(t - 1)]

[λ ​ x ​ ​ - V ​ x i ​ ​ (t - 1)]

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

Sutton & Barto 1981

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1999

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

Contiguity and LTP as the memory mechanism

Historical background

LTP as the memory mechanism

Aristotle 360 BC

David Hartley 1749

Edward C. Tolman 1948

Schukarew
1907

Bush &
Mosteller
1951

Kamin
1969

Wagner, Logan & Price
1968

Rescorla
1967

Rescorla Wagner
1972

Edward L.
Thorndike
1898

Estes
1950

Pavlov
1927

Clark L.
Hull
1935

Schultz
1997

Sutton & Barto
1981

Hawkins & Kandel
1984

1960 's and the fall of contiguity

Chomsky's Review of
B.F. Skinner's Verbal behavior
1967

Rumelhart
&
McClelland

1986

Parallel Distributed Processing:
Explorations in the Microstructure of Cognition

Historical background

millenia of learning theory

Historical background

Schultz, Dayan, Montague (1997) Science

Question

If Schultz is right which of the two protocols do the animal learns faster on?

Gallistel & King, 2009

Question

If Schultz is right which of the two protocols do the animal learns faster on?

Gallistel & King, 2009

....there's a fairly recent book by a very good cognitive neuroscientist, Randy Gallistel and King, arguing -- in my view, plausibly -- that neuroscience developed kind of enthralled to associationism....
- Chomsky
(Yarden Katz interview in Atlantic 2012)

Question

If Schultz is right which of the two protocols do the animal learns faster on?

Balsam & Gibbon, 1982

It is difficult to get a man to understand something, when his salary depends upon his not understanding it!
- Upton Sinclair (1935)

The Project

How to convince people of the inherent limitations of RL

betweenness as a spatial representation

Tarski (1956) proved that no binary relation of points can be used as fundamental primitives for obtaining Euclidean geometry.
A ternary relation is needed.

\text{Re}^N \text{ where } N \geq 1.

R e ​ N ​ ​ w h e r e N \geq 1 .

Experiment 1

Experiment 2

Experiment 3

Suh et al., (2011), Science (Tonegawa)

Experiment 4

8 sec.

12 sec.

20 sec.

Experiment 5

Experiment A

K.M. Taylor et al. / Behavioural Processes 101 (2014) 15–22

Experiment 6

Syt17-cre, 2x 0.5 microliter
AAV-DIO-Jaws-EGFP
AAV-DIO-ChR2-mCherry

Solution: tandem opsins

Kleinlogel, S., Terpitz, U., Legrum, B., Gükbuget, D., Boyden, E.S., Bamann, C., Wood, P.G. and Bamberg, E.: A gene-fusion strategy for stoichiometric and co-localized expression of light-gated membrane proteins.

Nat. Methods 8, 1083-1088. (2011).

GeneBank: JN836741

2A peptide cleaved ChR2 and NphR can be shuttled to different membrane locations and their different degradation rates can lead to variable stoichiometries of expression.
Transmembrane helix from the β subunit of the rat gastric H+,K+-ATPase

Solution: tandem opsins

Nat. Methods 8, 1083-1088. (2011).

GeneBank: JN836741