Temporal Integration in the Mesolimbic Pathway during Pavlovian Conditioning
Daniel Fürth, DMC lab meeting 8 december 2014
Content
- The Clinical Problem
- Historical background
- Walkthrough of the project
- Things to be done
Main Argument
- The brain does not compute running averages of past events (like reinforcement learning models).
- The brain has a rich temporal representation of events and their relation in time (a temporal map) just as rich as the spatial map.
- The organism gathers information about its environment and then makes decisions on grounds of this knowledge when it wants to.
- This view is essential to understand both aversive and appetitive learning.
Why I don't care much about computational neuroscience
When a man throws a ball high in the air and catches it again, he behaves as if he had solved a set of differential equations in predicting the trajectory of the ball... At some subconscious level, something functionally equivalent to the mathematical calculation is going on.
Richard Dawkins, Selfish Gene
m\frac{d^{2}x}{dt^{2}} + cS\left(\frac{dx}{dt}\right)^{2} = 0, \quad m\frac{d^{2}y}{dt^{2}} + cS\left(\frac{dy}{dt}\right)^{2} = -mg,
mdt2d2x+cS(dtdx)2=0,mdt2d2y+cS(dtdy)2=−mg,
\frac{d^{2}h_x}{dt^2} + k\left( \frac{dh_x}{dt} \right)^2 = 0, \quad h_x(t=0) = x_0 , \quad v_x = \left[\frac{dh_x}{dt}\right]_{t=0} = v_0cos(\alpha)
dt2d2hx+k(dtdhx)2=0,hx(t=0)=x0,vx=[dtdhx]t=0=v0cos(α)
\frac{d^{2}h_y}{dt^2} + k\left( \frac{dh_y}{dt} \right)^2 = -g, \quad h_y(t=0) = x_0 , \quad v_y = \left[\frac{dh_y}{dt}\right]_{t=0} = v_0sin(\alpha)
dt2d2hy+k(dtdhy)2=−g,hy(t=0)=x0,vy=[dtdhy]t=0=v0sin(α)
Gaze heuristic (McLeod et al. 2003, Nature)
Why I don't care much about computational neuroscience
linear optical trajectory (LOT) model (Shaffer et al. 2008)
Why I don't care much about computational neuroscience
To study the brain properly you should begin, like David Marr, by asking what tasks is it performing. Not assuming what it is doing.
The Clinical Problem
Treatment refusal (Issakidis & Andrews, 2004)
30%
Exposure based therapy.
Treatment attrition (Haby et al., 2006)
15-30%
Treatment non-responders (Loerencic et al. in submission)
40-50%
Return of fear (Craske & Mystkowski, 2006)
19-62%
11-29% of the intent-to-treat population will experience a substantial clinical effect
If return of fear solved: 25-36% about 400'000 individuals in Sweden.
If non-responders solved: 23-53% about 670'000 individuals.
Momentary prevalence of anxiety disorders: 12-17%.
The Clinical Problem
The Economical Problem
- How often should you meet your therapist?
- How many sessions should you schedule?
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
sufficient and necessary conditions for learning
- Similarity
- Contiguity
- Contrast
- Frequency
Laws of recall:
Did Aristotle actually claim this?
De Memoria Et Reminiscentia (Ross & Aristotle, 1906, p. 111)
Historical background
sufficient and necessary conditions for learning
- Similarity
- Contiguity
- Contrast
- Frequency
Laws of recall:
Did Aristotle actually claim this?
De Memoria Et Reminiscentia (Ross & Aristotle, 1906, p. 111)
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
learning as an autocatalytic set
Russian physicist Schukarew (1907) inspired by the law of physical chemistry known as the autocatalytic principle.
Some reactions accelerated by external substance, a catalyst.
Other autocatalyst - the products of the reaction itself accelerate the process.
- Amount of catalyst currently available.
- The amount of chemical as yet unreacted.
Autocatalytic Principle:
Rate of reaction is proportional to:
V_x(t)
Vx(t)
[\lambda_x - V_x(t)]
[λx−Vx(t)]
Historical background
learning as an autocatalytic set
\text{Rate at which learning occurs} = (\text{Amount of learning that has already occured}) \times (\text{Amount of learning that has yet to occur}),
Rate at which learning occurs=(Amount of learning that has already occured)×(Amount of learning that has yet to occur),
In words:
Expressed as a first-order differential equation:
\frac{\text{d}V}{\text{d}t} = \alpha \cdot V(\lambda - V),
dtdV=α⋅V(λ−V),
Where:
\alpha \text{ is a free learning rate parameter on the unit interval. } \alpha \in [0,1],
α is a free learning rate parameter on the unit interval. α∈[0,1],
Historical background
The shape of the learning curve
V_{xit} = \frac{\lambda_x}{(1 + \beta_x \text{exp}\{-\alpha_i \times TIME_t\})},
Vxit=(1+βxexp{−αi×TIMEt})λx,
\Delta V_{xi}(t) = \beta_x \alpha_i \cdot [\lambda_x - V_{xi}(t-1)]
ΔVxi(t)=βxαi⋅[λx−Vxi(t−1)]
Linear operator:
Logistic learning curve:
- Associative value / Response strength
- Salience of stimuli
- Learning rate for the individual.
- Prediction-error.
i\text{th}
ith
x
x
V_{xi}(t)
Vxi(t)
\beta_x
βx
\alpha_x
αx
[\lambda_x - V_{xi}(t - 1)]
[λx−Vxi(t−1)]
Historical background
learning as an autocatalytic set
Why autocatalytic processes?
Historical background
learning as an autocatalytic set
Why autocatalytic processes?
- At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.
Historical background
learning as an autocatalytic set
Why autocatalytic processes?
- At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.
-
This theory is ofcourse altogether abandonned:
-
Watson & Crick (1953), discovery of double helix
- Central Dogma of Mol. Biology.
-
Watson & Crick (1953), discovery of double helix
Historical background
learning as an autocatalytic set
Why autocatalytic processes?
- At the time it was thought that the synthesis of new proteins was made by autocatalytic processes similar to metabolism.
- e.g. one protein synthesis a new protein from itself.
-
This theory is ofcourse altogether abandonned:
-
Watson & Crick (1953), discovery of double helix
- Central Dogma of Mol. Biology.
-
Watson & Crick (1953), discovery of double helix
Assumption: proteins as the basis of memory (synaptic plasticity).
Concentration = Synaptic weight = Strength of memory trace
- No role for information.
- Compare autocatalysm to DNA and translation.
Historical background
engrams and memory traces
Engram (Karl Lashley, Richard Thompson etc.)
David Hasley (1746)
Forgetting (see Bjork, 2003)
- Trace decay (Thorndike, 1914)
- Consolidation (Müller & Pilzecker, 1900)
- Interference (Kahana, Brown, Lewandowsky)
Edward Throndike (1914) Law of Disuse
Consolidation: cannot explain proactive interference (old information that hinders retrieval of new information).
Decay theory: Completely destroyed by John McGeoch (1932). Mere passage of time doesn't lead to forgetting, what is done during this passage of time matters and can explain most.
Historical background
engrams and memory traces
Engram (Karl Lashley, Richard Thompson etc.)
David Hasley (1746)
Spike-Time-Dependent Plasticity (STDP) Markram et al. 1997
Edward Throndike (1914) Law of Disuse
RLT Credit assignment problem
Forgetting (see Bjork, 2003)
- Trace decay (Thorndike, 1914)
- Consolidation (Müller & Pilzecker, 1900)
- Interference (Kahana, Brown, Lewandowsky)
Historical background
memory traces and prediction errors
The Backward View of Temporal Difference (TD)-reinforcement learning
Midbrain dopaminergic nuclei as a finite state machine.
Historical background
engrams and memory traces
Crick, (1997) TRSB
Crick, (1984) Nature
Of course memories are not stored at synapses. But I think it is useful to contemplate the possibility that they are not stored anywhere else in the brain either.
The whole issue of where or, more important, how memories are stored in the brain may turn out to be an incorrect formulation of the problem, despite its seductively enticing allure. And the source of such an incorrect formulation may lie in the single-minded preoccupation with the storage, or the engram, and sometimes even identification of storage with memory.This preoccupation with the physical changes that follow from an experience that can be remembered seems to be accompanied by a rather conspicuous neglect of retrieval processes. - Endel Tulving
Historical background
engrams and memory traces
Historical background
engrams and memory traces
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
Tolman and Latent Learning
T-alley Maze
Tolman & Honzik, 1930
Phase I
Test
Tolman, Ritchie, and Kalish, 1946
Result
Historical background
Tolman's Cognitive Maps
Historical background
milleniums of learning theory
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
Bush & Mosteller, 1951
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
1960-70 contiguity hits the fan
Robert Rescorla's Truly random control.
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
1960-70 contiguity hits the fan
Wagner's Relative Validity.
Experimental group:
10 trials of Tone + Light with food.
10 trials of Click + Light without food
Control group:
5 trials of Tone + Light with food.
5 trials of Click + Light without food
5 trials of Tone + Light without food.
5 trials of Click + Light with food
Wagner, A.R., Logan, F.A., Haberlandt, K. & Price, T. (1968)
Historical background
1960-70 contiguity hits the fan
Wagner's Relative Validity.
Experimental group:
10 trials of Tone + Light with food.
10 trials of Click + Light without food
Control group:
5 trials of Tone + Light with food.
5 trials of Click + Light without food
5 trials of Tone + Light without food.
5 trials of Click + Light with food
Only conditioning to the light in the control group.
Wagner, A.R., Logan, F.A., Haberlandt, K. & Price, T. (1968)
The animal is attending more to a stimulus that constantly predicts the outcome and attending less to a poor predictor.
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
1960-70 contiguity hits the fan
Patricia Janak (2013) Nature Neuroscience
Kamin's Blocking Effect
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
Rescorla-Wagner model 1972
\Sigma V = V_A + V_B
ΣV=VA+VB
Backward blocking
\Delta V_{xi}(t) = \beta_x \alpha_i \cdot [\lambda_x - V_{xi}(t-1)]
ΔVxi(t)=βxαi⋅[λx−Vxi(t−1)]
- Associative value / Response strength
- Salience of stimuli
- Learning rate for the individual.
- Prediction-error.
i\text{th}
ith
x
x
V_{xi}(t)
Vxi(t)
\beta_x
βx
\alpha_x
αx
[\lambda_x - V_{xi}(t - 1)]
[λx−Vxi(t−1)]
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
Sutton & Barto 1981
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1999
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
Contiguity and LTP as the memory mechanism
Historical background
LTP as the memory mechanism
Aristotle 360 BC
David Hartley 1749
Edward C. Tolman 1948
Schukarew
1907
Bush &
Mosteller
1951
Kamin
1969
Wagner, Logan & Price
1968
Rescorla
1967
Rescorla Wagner
1972
Edward L.
Thorndike
1898
Estes
1950
Pavlov
1927
Clark L.
Hull
1935
Schultz
1997
Sutton & Barto
1981
Hawkins & Kandel
1984
1960 's and the fall of contiguity
Chomsky's Review of
B.F. Skinner's Verbal behavior
1967
Rumelhart
&
McClelland
1986
Parallel Distributed Processing:
Explorations in the Microstructure of Cognition
Historical background
millenia of learning theory
Historical background
Schultz, Dayan, Montague (1997) Science
Question
If Schultz is right which of the two protocols do the animal learns faster on?
Gallistel & King, 2009
Question
If Schultz is right which of the two protocols do the animal learns faster on?
Gallistel & King, 2009
....there's a fairly recent book by a very good cognitive neuroscientist, Randy Gallistel and King, arguing -- in my view, plausibly -- that neuroscience developed kind of enthralled to associationism....
- Chomsky
(Yarden Katz interview in Atlantic 2012)
Question
If Schultz is right which of the two protocols do the animal learns faster on?
Balsam & Gibbon, 1982
It is difficult to get a man to understand something, when his salary depends upon his not understanding it!
- Upton Sinclair (1935)
The Project
How to convince people of the inherent limitations of RL
betweenness as a spatial representation
Tarski (1956) proved that no binary relation of points can be used as fundamental primitives for obtaining Euclidean geometry.
A ternary relation is needed.
\text{Re}^N \text{ where } N \geq 1.
ReN where N≥1.
Experiment 1
Experiment 2
Experiment 2
Experiment 3
Suh et al., (2011), Science (Tonegawa)
Experiment 4
8 sec.
12 sec.
20 sec.
Experiment 5
Experiment A
K.M. Taylor et al. / Behavioural Processes 101 (2014) 15–22
Experiment 6
Experiment 6
Experiment 6
Experiment 6
Experiment 6
Experiment 6
Experiment 6
Experiment 6
Syt17-cre, 2x 0.5 microliter
AAV-DIO-Jaws-EGFP
AAV-DIO-ChR2-mCherry
Solution: tandem opsins
Kleinlogel, S., Terpitz, U., Legrum, B., Gükbuget, D., Boyden, E.S., Bamann, C., Wood, P.G. and Bamberg, E.: A gene-fusion strategy for stoichiometric and co-localized expression of light-gated membrane proteins.
Nat. Methods 8, 1083-1088. (2011).
GeneBank: JN836741
- 2A peptide cleaved ChR2 and NphR can be shuttled to different membrane locations and their different degradation rates can lead to variable stoichiometries of expression.
- Transmembrane helix from the β subunit of the rat gastric H+,K+-ATPase
Solution: tandem opsins
Kleinlogel, S., Terpitz, U., Legrum, B., Gükbuget, D., Boyden, E.S., Bamann, C., Wood, P.G. and Bamberg, E.: A gene-fusion strategy for stoichiometric and co-localized expression of light-gated membrane proteins.
Nat. Methods 8, 1083-1088. (2011).
GeneBank: JN836741
deck
By Daniel Fürth
