From Linear Systems to NNs
Aadharsh Aadhithya A
3x+4y=12
3x+4y=12
What does it mean?
3x+4y=12
3x+4y=12
What does it mean?
Set of all points (x,y) that satisfy the relation 3x+4y=12.
3x+4y=12
3x+4y=12
What does it mean?
Set of all points (x,y) that satisfy the relation 3x+4y=12.
How many soltions possible?
3x+4y=12
3x+4y=12
What does it mean?
Set of all points (x,y) that satisfy the relation 3x+4y=12.
How many soltions possible?
∞
\infty
3x+4y=12
3x+4y=12
Set of all points (x,y) that satisfy the relation 3x+4y=12.
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
3x+4y=12
3x+4y=12
Set of all points (x,y) that satisfy the relation 3x+4y=12.
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
cTx=b
c^T x = b
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin? r
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin?
But what is the direction perpendicular to that line?
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin?
But what is the direction perpendicular to that line?
y=−43x+3
y = -\frac{3}{4}x+3
y=mx+c
y=mx+c
m=ΔxΔy
m = \frac{\Delta y}{\Delta x}
4
3
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin?
But what is the direction perpendicular to that line?
y=−43x+3
y = -\frac{3}{4}x+3
y=mx+c
y=mx+c
m=ΔxΔy
m = \frac{\Delta y}{\Delta x}
4
3
[4−3]
\begin{bmatrix}
4 \\ -3
\end{bmatrix}
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin?
But what is the direction perpendicular to that line?
4
3
[4−3]
\begin{bmatrix}
4 \\ -3
\end{bmatrix}
A vector that is perpendicular to [4 3]
[4−3]⋅v=0
\begin{bmatrix}
4\\
-3
\end{bmatrix}
\cdot
v
=
0
v=[34]
v = \begin{bmatrix}
3\\
4
\end{bmatrix}
[34]
\begin{bmatrix}
3 \\ 4
\end{bmatrix}
Now, From these infinite solutions, give me a solution that is closest to the origin!
[34][xy]=12
\begin{bmatrix}
3 & 4
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix} = 12
The solution lies on the perpendicular line from the origin?
4
3
[43]
\begin{bmatrix}
4 \\ 3
\end{bmatrix}
[34]
\begin{bmatrix}
3 \\ 4
\end{bmatrix}
Then This solution is of the form
c⋅[34]
c \cdot \begin{bmatrix}
3 \\ 4 \end{bmatrix}
Excercise: Find c
Least Norm Solution
Now, From these infinite solutions, give me a solution that is closest to the origin!
Least Norm Solution
[345]xyz=12
\begin{bmatrix}
3 & 4 & 5
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = 12
This is a plane in 3D
Now, From these infinite solutions, give me a solution that is closest to the origin!
Least Norm Solution
[345]xyz=12
\begin{bmatrix}
3 & 4 & 5
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = 12
This is a plane in 3D
Infinite solutions are possible. "Least-Norm" Solution is again in a direction perpendicular to this plane
Now, From these infinite solutions, give me a solution that is closest to the origin!
Least Norm Solution
[345]xyz=12
\begin{bmatrix}
3 & 4 & 5
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = 12
This is a plane in 3D
Infinite solutions are possible. "Least-Norm" Solution is again in a direction perpendicular to this plane
Direction that is perpendicular to the plane is:
345
\begin{bmatrix} 3 \\ 4 \\ 5
\end{bmatrix}
Now, From these infinite solutions, give me a solution that is closest to the origin!
[345]xyz=12
\begin{bmatrix}
3 & 4 & 5
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = 12
This is a plane in 3D
Infinite solutions are possible. "Least-Norm" Solution is again in a direction perpendicular to this plane
c⋅345
c \cdot \begin{bmatrix} 3 \\ 4 \\ 5
\end{bmatrix}
Least norm solution is of the form
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Intersection of two planes: is a line
Infinite solutions!
Infinite points. Find point, closest to the origin!
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Intersection of two planes: is a line
Infinite solutions!
Infinite points. Find point, closest to the origin!
The Point is on a line Orthogonal to the Line of intersection of both the planes.
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Intersection of two planes: is a line
Infinite solutions!
Infinite points. Find point, closest to the origin!
The Point is on a line Orthogonal to the Line of intersection of both the planes.
But But, There are Skew lines in 3D!
But it should be on the plane perpendicular to the line
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Intersection of two planes: is a line
Infinite solutions!
Infinite points. Find point, closest to the origin!
But But, There are Skew lines in 3D!
But it should be on the plane perpendicular to the line
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
What does this mean geometrically?
Intersection of two planes: is a line
Infinite solutions!
Infinite points. Find point, closest to the origin!
But But, There are Skew lines in 3D!
But it should be on the plane perpendicular to the line
c1⋅345+c2⋅678
c_1 \cdot \begin{bmatrix} 3 \\4\\5
\end{bmatrix}
+
c_2 \cdot \begin{bmatrix} 6 \\7\\8
\end{bmatrix}
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
Infinite solutions!
c1⋅345+c2⋅678
c_1 \cdot \begin{bmatrix} 3 \\4\\5
\end{bmatrix}
+
c_2 \cdot \begin{bmatrix} 6 \\7\\8
\end{bmatrix}
LESSON
The least norm solution, always lies in the row space of A
Now, From these infinite solutions, give me a solution that is closest to the origin!
[364758]xyz=[1213]
\begin{bmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
\end{bmatrix}
\begin{bmatrix}
x \\ y \\ z
\end{bmatrix} = \begin{bmatrix}
12 \\
13
\end{bmatrix}
Infinite solutions!
c1⋅345+c2⋅678
c_1 \cdot \begin{bmatrix} 3 \\4\\5
\end{bmatrix}
+
c_2 \cdot \begin{bmatrix} 6 \\7\\8
\end{bmatrix}
Theorem
The least norm solution, always lies in the row space of A
Representer Theorem
Ax=b
Ax=b
The least norm solution lies in the rowspace of A. In other words, it is a linear combination of the rows of A
ATc
A^Tc
It is of the form
Ax=b
Ax=b
The least norm solution lies in the rowspace of A. In other words, it is a linear combination of the rows of A
ATc
A^Tc
It is of the form
A(ATc)=b
A(A^T c) = b
x=(AAT)−1b
x = (AA^T)^{-1} b
Ax=b
Ax=b
The least norm solution lies in the rowspace of A. In other words, it is a linear combination of the rows of A
ATc
A^Tc
It is of the form
A(ATc)=b
A(A^T c) = b
x=(AAT)−1b
x = (AA^T)^{-1} b
What if AA^T is not invertable????
Resort to Pinv
Pseudoinverse
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Column Space
LNS
Row Space
Right Null Space
A is 8x8, with rank 4
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
UT(UΣVT)x=UTb
U^T(U \Sigma V^T) x= U^T b
Column Space
LNS
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
UT(UΣVT)x=UTb
U^T(U \Sigma V^T) x= U^T b
Column Space
LNS
Column Space
LNS
B In Col Space
B Not In Col Space
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
UT(UΣVT)x=UTb
U^T(U \Sigma V^T) x= U^T b
Column Space
LNS
Column Space
LNS
B In Col Space
B Not In Col Space
=
=
=
=
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
IΣVTx=UTb
I \Sigma V^T x= U^T b
Column Space
LNS
Column Space
LNS
B In Col Space
B Not In Col Space
=
=
=
=
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
ΣVTx=UTb
\Sigma V^T x= U^T b
Σ=σ100⋮00000σ20⋮⋯00000σ3⋱⋯000⋯⋯⋯⋱⋯⋯⋯⋯000⋮σr⋯⋯⋯000⋮⋯⋯⋯⋯000⋮0000000⋮0000000⋮0000
\Sigma = \begin{bmatrix}
\sigma_1 & 0 & 0 & \cdots & 0 & 0 & 0 & 0 & 0 \\
0 & \sigma_2 & 0 & \cdots & 0 & 0& 0 & 0 & 0 \\
0 & 0 & \sigma_3 & \cdots & 0 & 0& 0 & 0 & 0 \\
\vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\
0 & \cdots & \cdots & \cdots& \sigma_r & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
\end{bmatrix}
Σ†=σ1100⋮00000σ210⋮⋯00000σ31⋱⋯000⋯⋯⋯⋱⋯⋯⋯⋯000⋮σr1⋯⋯⋯000⋮⋯⋯⋯⋯000⋮0000000⋮0000000⋮0000
\Sigma^{\dagger} = \begin{bmatrix}
\frac{1}{\sigma_1} & 0 & 0 & \cdots & 0 & 0 & 0 & 0 & 0 \\
0 & \frac{1}{\sigma_2} & 0 & \cdots & 0 & 0& 0 & 0 & 0 \\
0 & 0 & \frac{1}{\sigma_3} & \cdots & 0 & 0& 0 & 0 & 0 \\
\vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\
0 & \cdots & \cdots & \cdots& \frac{1}{\sigma_r} & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
\end{bmatrix}
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
VTx=Σ†(UTb)
V^T x= \Sigma^{\dagger}(U^T b)
Σ†=σ1100⋮000000σ210⋮⋯000000σ31⋱⋯0000⋯⋯⋯⋱⋯⋯⋯⋯⋯000⋮σr1⋯⋯⋯⋯000⋮⋯⋯⋯⋯⋯000⋮00000000⋮00000000⋮00000
\Sigma^{\dagger} = \begin{bmatrix}
\frac{1}{\sigma_1} & 0 & 0 & \cdots & 0 & 0 & 0 & 0 & 0 \\
0 & \frac{1}{\sigma_2} & 0 & \cdots & 0 & 0& 0 & 0 & 0 \\
0 & 0 & \frac{1}{\sigma_3} & \cdots & 0 & 0& 0 & 0 & 0 \\
\vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\
0 & \cdots & \cdots & \cdots& \frac{1}{\sigma_r} & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
\end{bmatrix}
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
VTx=Σ†(UTb)
V^T x= \Sigma^{\dagger}(U^T b)
Σ†=σ1100⋮000000σ210⋮⋯000000σ31⋱⋯0000⋯⋯⋯⋱⋯⋯⋯⋯⋯000⋮σr1⋯⋯⋯⋯000⋮⋯⋯⋯⋯⋯000⋮00000000⋮00000000⋮00000
\Sigma^{\dagger} = \begin{bmatrix}
\frac{1}{\sigma_1} & 0 & 0 & \cdots & 0 & 0 & 0 & 0 & 0 \\
0 & \frac{1}{\sigma_2} & 0 & \cdots & 0 & 0& 0 & 0 & 0 \\
0 & 0 & \frac{1}{\sigma_3} & \cdots & 0 & 0& 0 & 0 & 0 \\
\vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\
0 & \cdots & \cdots & \cdots& \frac{1}{\sigma_r} & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
\end{bmatrix}
+
+
+
+
+
+
+
+
+
+
+
+
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
VTx=Σ†(UTb)
V^T x= \Sigma^{\dagger}(U^T b)
Σ†=σ1100⋮000000σ210⋮⋯000000σ31⋱⋯0000⋯⋯⋯⋱⋯⋯⋯⋯⋯000⋮σr1⋯⋯⋯⋯000⋮⋯⋯⋯⋯⋯000⋮00000000⋮00000000⋮00000
\Sigma^{\dagger} = \begin{bmatrix}
\frac{1}{\sigma_1} & 0 & 0 & \cdots & 0 & 0 & 0 & 0 & 0 \\
0 & \frac{1}{\sigma_2} & 0 & \cdots & 0 & 0& 0 & 0 & 0 \\
0 & 0 & \frac{1}{\sigma_3} & \cdots & 0 & 0& 0 & 0 & 0 \\
\vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\
0 & \cdots & \cdots & \cdots& \frac{1}{\sigma_r} & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
0 & 0 & 0 & \cdots& \cdots & \cdots & 0 & 0 & 0 \\
\end{bmatrix}
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
VTx=Σ†(UTb)
V^T x= \Sigma^{\dagger}(U^T b)
Row Space
Right Null Space
=
=
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
x=Vӆ(UTb)
x= V \Sigma^{\dagger}(U^T b)
=
=
Row Space
Right Null Space
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
x=Vӆ(UTb)
x= V \Sigma^{\dagger}(U^T b)
=
=
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
x=Vӆ(UTb)
x= V \Sigma^{\dagger}(U^T b)
=
=
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Your Solution is a linear combination of Row Space
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
x=Vӆ(UTb)
x= V \Sigma^{\dagger}(U^T b)
=
=
Pinv Gives: Least Norm Solution
Pseudoinverse
A=UΣVT
A = U \Sigma V^T
Ax=b
Ax=b
(UΣVT)x=b
(U \Sigma V^T) x=b
UTU=UUT=I
U^TU=UU^T = I
x=Vӆ(UTb)
x= V \Sigma^{\dagger}(U^T b)
=
=
Pinv Gives: Least Norm Solution
If b is not in col space, It gives Least Squares and Least Norm solution
Linear Regression
Linear Regression
y=mx+c
y=mx+c
(xi,yi)
(x_i,y_i)
x1x2x3⋮xn1111⋅[mc]=y1y2y3⋮yn
\begin{bmatrix}
x_1 & 1 \\
x_2 & 1 \\
x_3 & 1 \\
\vdots \\
x_n & 1 \\
\end{bmatrix} \cdot
\begin{bmatrix} m\\ c \end{bmatrix} =
\begin{bmatrix}
y_1\\
y_2 \\
y_3 \\
\vdots \\
y_n \\
\end{bmatrix}
Linear Regression
y=mx+c
y=mx+c
(xi,yi)
(x_i,y_i)
x1x2x3⋮xn1111⋅[mc]=y1y2y3⋮yn
\begin{bmatrix}
x_1 & 1 \\
x_2 & 1 \\
x_3 & 1 \\
\vdots \\
x_n & 1 \\
\end{bmatrix} \cdot
\begin{bmatrix} m\\ c \end{bmatrix} =
\begin{bmatrix}
y_1\\
y_2 \\
y_3 \\
\vdots \\
y_n \\
\end{bmatrix}
wmin∣∣y−wTx∣∣
\min_w|| y - w^Tx||
i=1∑N(yi−wxi)2
\sum_{i=1}^N (y_i - w x_i)^2
Xw=Y
X w = Y
Linear Regression
y=mx+c
y=mx+c
(xi,yi)
(x_i,y_i)
x1x2x3⋮xn1111⋅[mc]≈y1y2y3⋮yn
\begin{bmatrix}
x_1 & 1 \\
x_2 & 1 \\
x_3 & 1 \\
\vdots \\
x_n & 1 \\
\end{bmatrix} \cdot
\begin{bmatrix} m\\ c \end{bmatrix} \approx
\begin{bmatrix}
y_1\\
y_2 \\
y_3 \\
\vdots \\
y_n \\
\end{bmatrix}
wmin∣∣Y−wTX∣∣22
\min_w|| Y - w^T X||_2^2
i=1∑N(yi−wxi)2
\sum_{i=1}^N (y_i - w x_i)^2
Xw≈Y
X w \approx Y
w=X†Y
w =X^{\dagger} Y
Kernel Regression
Kernel Regression
Perform Linear Regression, after Mapping Data
https://xavierbourretsicotte.github.io/Kernel_feature_map.html
Kernel Regression
Perform Linear Regression, after Mapping Data
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−wxi)2
L(w) = \sum_{i=1}^N (y_i - w x_i)^2
L(w)=i=1∑N(yi−w⋅ψ(xi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot \psi(x_i))^2
Linear Regression
Kernel Regression
Kernel Regression
Perform Linear Regression, after Mapping Data
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−wxi)2
L(w) = \sum_{i=1}^N (y_i - w x_i)^2
L(w)=i=1∑N(yi−w⋅ψ(xi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot \psi(x_i))^2
Linear Regression
Kernel Regression
Typically we map to higher dimentions, i.e k>d
Rk
\mathbb{R}^k
Kernel Regression
Perform Linear Regression, after Mapping Data
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−wxi)2
L(w) = \sum_{i=1}^N (y_i - w x_i)^2
L(w)=i=1∑N(yi−w⋅ψ(xi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot \psi(x_i))^2
Linear Regression
Kernel Regression
Rk
\mathbb{R}^k
We have k unknowns
Kernel Regression
Perform Linear Regression, after Mapping Data
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−w⋅ψ(xi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot \psi(x_i))^2
Least norm solution,
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(α)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(\alpha) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
α1ψ(x1)ψ(xi)+α2ψ(x2)ψ(xi)+α3ψ(x3)ψ(xi)+⋯αnψ(xn)ψ(xi)+
\alpha_1 \psi(x_1) \psi(x_i) + \\
\alpha_2 \psi(x_2) \psi(x_i) + \\
\alpha_3 \psi(x_3) \psi(x_i) + \\
\cdots\\
\alpha_n \psi(x_n) \psi(x_i) +
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
α1ψ(x1)ψ(xi)+α2ψ(x2)ψ(xi)+α3ψ(x3)ψ(xi)+⋯αnψ(xn)ψ(xi)+
\alpha_1 \psi(x_1) \psi(x_i) + \\
\alpha_2 \psi(x_2) \psi(x_i) + \\
\alpha_3 \psi(x_3) \psi(x_i) + \\
\cdots\\
\alpha_n \psi(x_n) \psi(x_i) +
[α1α2α3⋯αn]ψ(x1)⋅ψ(xi)ψ(x2)⋅ψ(xi)ψ(x3)⋅ψ(xi)⋯ψ(xn)⋅ψ(xi)
\begin{bmatrix} \alpha_1 & \alpha_2 & \alpha_3 \cdots \alpha_n \end{bmatrix}\begin{bmatrix}
\psi(x_1) \cdot \psi(x_i) \\
\psi(x_2) \cdot \psi(x_i) \\
\psi(x_3) \cdot \psi(x_i) \\
\cdots\\
\psi(x_n) \cdot \psi(x_i)
\end{bmatrix}
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
[α1α2α3⋯αn]ψ(x1)⋅ψ(xi)ψ(x2)⋅ψ(xi)ψ(x3)⋅ψ(xi)⋯ψ(xn)⋅ψ(xi)
\begin{bmatrix} \alpha_1 & \alpha_2 & \alpha_3 \cdots \alpha_n \end{bmatrix}\begin{bmatrix}
\psi(x_1) \cdot \psi(x_i) \\
\psi(x_2) \cdot \psi(x_i) \\
\psi(x_3) \cdot \psi(x_i) \\
\cdots\\
\psi(x_n) \cdot \psi(x_i)
\end{bmatrix}
αT
\alpha^T
K(X,xi)
K(X,x_i)
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
[α1α2α3⋯αn]ψ(x1)⋅ψ(xi)ψ(x2)⋅ψ(xi)ψ(x3)⋅ψ(xi)⋯ψ(xn)⋅ψ(xi)
\begin{bmatrix} \alpha_1 & \alpha_2 & \alpha_3 \cdots \alpha_n \end{bmatrix}\begin{bmatrix}
\psi(x_1) \cdot \psi(x_i) \\
\psi(x_2) \cdot \psi(x_i) \\
\psi(x_3) \cdot \psi(x_i) \\
\cdots\\
\psi(x_n) \cdot \psi(x_i)
\end{bmatrix}
αT
\alpha^T
K(X,xi)
K(X,x_i)
L(α)=i=1∑N(yi−αTK(X,xi))2
L(\alpha) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
[α1α2α3⋯αn]ψ(x1)⋅ψ(xi)ψ(x2)⋅ψ(xi)ψ(x3)⋅ψ(xi)⋯ψ(xn)⋅ψ(xi)
\begin{bmatrix} \alpha_1 & \alpha_2 & \alpha_3 \cdots \alpha_n \end{bmatrix}\begin{bmatrix}
\psi(x_1) \cdot \psi(x_i) \\
\psi(x_2) \cdot \psi(x_i) \\
\psi(x_3) \cdot \psi(x_i) \\
\cdots\\
\psi(x_n) \cdot \psi(x_i)
\end{bmatrix}
αT
\alpha^T
K(X,xi)
K(X,x_i)
L(w)=i=1∑N(yi−αTK(X,xi))2
L(w) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
N unknowns
Even if we map to "infinite dimensions", we still have to determine only N unknowns
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
L(w)=i=1∑N(yi−αTK(X,xi))2
L(w) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
This is exactly Linear regression over alpha
K^=ψ(x1)ψ(x1)ψ(x2)ψ(x1)⋮ψ(xn)ψ(x1)ψ(x1)ψ(x2)ψ(x2)ψ(x2)⋮ψ(xn)ψ(xn)⋯⋯⋱⋯ψ(x1)ψ(xn)ψ(x2)ψ(xn)⋮ψ(xn)ψ(xn)
\hat{K} = \begin{bmatrix}
\psi(x_1) \psi(x_1) & \psi(x_1) \psi(x_2) & \cdots & \psi(x_1) \psi(x_n) \\
\psi(x_2) \psi(x_1) & \psi(x_2) \psi(x_2) & \cdots & \psi(x_2) \psi(x_n) \\
\vdots & \vdots & \ddots & \vdots \\
\psi(x_n) \psi(x_1) & \psi(x_n) \psi(x_n) & \cdots & \psi(x_n) \psi(x_n) \\
\end{bmatrix}
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
L(w)=i=1∑N(yi−αTK(X,xi))2
L(w) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
This is exactly Linear regression over alpha
K^=ψ(x1)ψ(x1)ψ(x2)ψ(x1)⋮ψ(xn)ψ(x1)ψ(x1)ψ(x2)ψ(x2)ψ(x2)⋮ψ(xn)ψ(xn)⋯⋯⋱⋯ψ(x1)ψ(xn)ψ(x2)ψ(xn)⋮ψ(xn)ψ(xn)
\hat{K} = \begin{bmatrix}
\psi(x_1) \psi(x_1) & \psi(x_1) \psi(x_2) & \cdots & \psi(x_1) \psi(x_n) \\
\psi(x_2) \psi(x_1) & \psi(x_2) \psi(x_2) & \cdots & \psi(x_2) \psi(x_n) \\
\vdots & \vdots & \ddots & \vdots \\
\psi(x_n) \psi(x_1) & \psi(x_n) \psi(x_n) & \cdots & \psi(x_n) \psi(x_n) \\
\end{bmatrix}
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
L(w)=i=1∑N(yi−αTK(X,xi))2
L(w) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
This is exactly Linear regression over alpha
αK^=y
\alpha \hat{K} = y
α=yK^†
\alpha = y \hat{K}^{\dagger}
Kernel Regression
ψ:Rd→Rk
\psi: \mathbb{R}^d \rightarrow \mathbb{R}^k
L(w)=i=1∑N(yi−(j∑Nαjψ(xj))⋅ψ(xi))2
L(w) = \sum_{i=1}^N \left (y_i - \left ( \sum_j^{N} \alpha_j \psi(x_j) \right ) \cdot \psi(x_i) \right)^2
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
L(w)=i=1∑N(yi−αTK(X,xi))2
L(w) = \sum_{i=1}^N \left (y_i - \alpha^T K(X,x_i) \right)^2
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
This is exactly Linear regression over alpha
αK^=y
\alpha \hat{K} = y
α=yK^†
\alpha = y \hat{K}^{\dagger}
w=j∑Nαjψ(xj)
w = \sum_j^{N} \alpha_j \psi(x_j)
f(x)=yK†K(X,x)
f(x) = y K^{\dagger} K(X,x)
NNGP
Neural Network Gaussian Process
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
f(x)=Aϕ(B⋅x)
f(x) = A \phi(B \cdot x)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
f(x)=Aϕ(B⋅x)
f(x) = A \phi(B \cdot x)
Rd
\mathbb{R}^d
Rk×d
\mathbb{R}^{k \times d}
R1×k
\mathbb{R}^{1 \times k}
k
k
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
f(x)=Aϕ(B⋅x)
f(x) = A \phi(B \cdot x)
k
k
L(w)=i=1∑N(yi−w⋅Aϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot A\phi(Bx_i))^2
Mapping x to R^k
B is initialized Randomly
Fix B, Update A only
Bij∼N(0,1)
B_{ij} \sim \mathcal{N}(0,1)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
f(x)=Aϕ(B⋅x)
f(x) = A \phi(B \cdot x)
L(w)=i=1∑N(yi−Aϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A\phi(Bx_i))^2
Map, do Linear Regression
Bij∼N(0,1)
B_{ij} \sim \mathcal{N}(0,1)
i.e Kernel Regression
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
αK^=y
\alpha \hat{K} = y
α=yK^†
\alpha = y \hat{K}^{\dagger}
k→∞lim
\lim_{k\rightarrow \infty}
Aϕ(B⋅x)
A \phi(B \cdot x)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
L(w)=i=1∑N(yi−w⋅Aϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - w \cdot A\phi(Bx_i))^2
Map, do Linear Regression
Bij∼N(0,1)
B_{ij} \sim \mathcal{N}(0,1)
i.e Kernel Regression
L(w)=∣∣y−αK^∣∣22
L(w) = || y - \alpha \hat{K} ||_2^2
αK^=y
\alpha \hat{K} = y
α=yK^†
\alpha = y \hat{K}^{\dagger}
k→∞lim
\lim_{k\rightarrow \infty}
Aϕ(B⋅x)
A \phi(B \cdot x)
How to calculate the kernel matrix K?
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
L(w)=i=1∑N(yi−A⋅k1ϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A \cdot \frac{1}{\sqrt{k}}\phi(Bx_i))^2
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
K^=ψ(x1)ψ(x1)ψ(x2)ψ(x1)⋮ψ(xn)ψ(x1)ψ(x1)ψ(x2)ψ(x2)ψ(x2)⋮ψ(xn)ψ(xn)⋯⋯⋱⋯ψ(x1)ψ(xn)ψ(x2)ψ(xn)⋮ψ(xn)ψ(xn)
\hat{K} = \begin{bmatrix}
\psi(x_1) \psi(x_1) & \psi(x_1) \psi(x_2) & \cdots & \psi(x_1) \psi(x_n) \\
\psi(x_2) \psi(x_1) & \psi(x_2) \psi(x_2) & \cdots & \psi(x_2) \psi(x_n) \\
\vdots & \vdots & \ddots & \vdots \\
\psi(x_n) \psi(x_1) & \psi(x_n) \psi(x_n) & \cdots & \psi(x_n) \psi(x_n) \\
\end{bmatrix}
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
L(w)=i=1∑N(yi−A⋅k1ϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A \cdot \frac{1}{\sqrt{k}}\phi(Bx_i))^2
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
K^=ψ(x1)ψ(x1)ψ(x2)ψ(x1)⋮ψ(xn)ψ(x1)ψ(x1)ψ(x2)ψ(x2)ψ(x2)⋮ψ(xn)ψ(xn)⋯⋯⋱⋯ψ(x1)ψ(xn)ψ(x2)ψ(xn)⋮ψ(xn)ψ(xn)
\hat{K} = \begin{bmatrix}
\psi(x_1) \psi(x_1) & \psi(x_1) \psi(x_2) & \cdots & \psi(x_1) \psi(x_n) \\
\psi(x_2) \psi(x_1) & \psi(x_2) \psi(x_2) & \cdots & \psi(x_2) \psi(x_n) \\
\vdots & \vdots & \ddots & \vdots \\
\psi(x_n) \psi(x_1) & \psi(x_n) \psi(x_n) & \cdots & \psi(x_n) \psi(x_n) \\
\end{bmatrix}
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=⟨k1ϕ(B⋅x),k1ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \left \langle \frac{1}{\sqrt{k}} \phi(B \cdot x), \frac{1}{\sqrt{k}} \phi(B \cdot \tilde{x}) \right \rangle
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
L(w)=i=1∑N(yi−A⋅k1ϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A \cdot \frac{1}{\sqrt{k}}\phi(Bx_i))^2
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=⟨k1ϕ(B⋅x),k1ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \left \langle \frac{1}{\sqrt{k}} \phi(B \cdot x), \frac{1}{\sqrt{k}} \phi(B \cdot \tilde{x}) \right \rangle
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
L(w)=i=1∑N(yi−A⋅k1ϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A \cdot \frac{1}{\sqrt{k}}\phi(Bx_i))^2
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=⟨k1ϕ(B⋅x),k1ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \left \langle \frac{1}{\sqrt{k}} \phi(B \cdot x), \frac{1}{\sqrt{k}} \phi(B \cdot \tilde{x}) \right \rangle
K(x,x~)=k1⟨ϕ(B⋅x),ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \frac{1}{k} \left \langle \phi(B \cdot x), \phi(B \cdot \tilde{x}) \right \rangle
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x~)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot \tilde{x})
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
L(w)=i=1∑N(yi−A⋅k1ϕ(Bxi))2
L(w) = \sum_{i=1}^N (y_i - A \cdot \frac{1}{\sqrt{k}}\phi(Bx_i))^2
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=⟨k1ϕ(B⋅x),k1ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \left \langle \frac{1}{\sqrt{k}} \phi(B \cdot x), \frac{1}{\sqrt{k}} \phi(B \cdot \tilde{x}) \right \rangle
K(x,x~)=k1⟨ϕ(B⋅x),ϕ(B⋅x~)⟩
K(x,\tilde{x}) = \frac{1}{k} \left \langle \phi(B \cdot x), \phi(B \cdot \tilde{x}) \right \rangle
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
DETOUR: SAMPLING FROM DISTRIBUTIONS AND ESTIMATION OF MEAN
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
These are samples from Normal Distributions
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
These are samples from Normal Distributions
We are summing over Samples from "a function of a Random Variable" and taking average
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
These are samples from Normal Distributions
We are summing over Samples from "a function of a Random Variable" and taking average
As we take larger and larger k, the more closer estimate of the mean/Expectation we get
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
As we take larger and larger k, the more closer estimate of the mean/Expectation we get
wT
w^T
wT
w^T
w∼N(0,I)
w \sim \mathcal{N}(0,I)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
As we take larger and larger k, the more closer estimate of the mean/Expectation we get
wT
w^T
wT
w^T
w∼N(0,I)
w \sim \mathcal{N}(0,I)
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
As we take larger and larger k, the more closer estimate of the mean/Expectation we get
wT
w^T
wT
w^T
w∼N(0,I)
w \sim \mathcal{N}(0,I)
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=Ew∼N(0,1)[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(u) \phi(v) \right ]
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u and v are from different distributions
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u and v are from different distributions
u=wTx
u = w^Tx
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u and v are from different distributions
u=wTx
u = w^Tx
E[u]=E[w1x1+w2x2⋯wdxd]
\mathbb{E}[u] = \mathbb{E}[w_1x_1 + w_2x_2 \cdots w_d x_d]
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u=wTx
u = w^Tx
E[u]=E[w1x1+w2x2⋯wdxd]
\mathbb{E}[u] = \mathbb{E}[w_1x_1 + w_2x_2 \cdots w_d x_d]
E[u]=E[w1x1]+E[w2x2]⋯E[wdxd]=0
\mathbb{E}[u] = \mathbb{E}[w_1x_1 ] + \mathbb{E}[w_2x_2] \cdots \mathbb{E}[w_d x_d] = 0
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u=wTx
u = w^Tx
E[u]=E[w1x1+w2x2⋯wdxd]
\mathbb{E}[u] = \mathbb{E}[w_1x_1 + w_2x_2 \cdots w_d x_d]
E[u]=E[w1x1]+E[w2x2]⋯E[wdxd]=0
\mathbb{E}[u] = \mathbb{E}[w_1x_1 ] + \mathbb{E}[w_2x_2] \cdots \mathbb{E}[w_d x_d] = 0
E[v]=0
\mathbb{E}[v] = 0
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u=wTx
u = w^Tx
E[v]=0
\mathbb{E}[v] = 0
E[u]=0
\mathbb{E}[u] = 0
Var(u)=Var(w1x1+w2x2⋯wdxd)=x12Var(w1)+x22Var(w2)⋯xd2Var(wd)
Var(u) = Var(w_1x_1 + w_2x_2 \cdots w_d x_d) = x_1^2 Var(w_1) + x_2^2 Var(w_2) \cdots x_d^2 Var(w_d)
Var(u)=x12+x22⋯xd2=∣∣x∣∣22
Var(u) = x_1^2 + x_2^2 \cdots x_d^2 = ||x||_2^2
cov(u,v)=⟨x,x~⟩
cov(u,v) = \langle x, \tilde{x} \rangle
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
E[v]=0
\mathbb{E}[v] = 0
E[u]=0
\mathbb{E}[u] = 0
Var(u)=∣∣x∣∣22
Var(u) = ||x||_2^2
Var(v)=∣∣x~∣∣22
Var(v) = ||\tilde{x}||_2^2
cov(u,v)=⟨x,x~⟩
cov(u,v) = \langle x,\tilde{x} \rangle
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u,v∼N(0,Λ)
u,v \sim \mathcal{N}(0,\Lambda)
Λ=[∣∣x∣∣22x⋅xTx⋅xT∣∣x~∣∣22]
\Lambda = \begin{bmatrix}
||x||_2^2 & x \cdot x^T \\
x \cdot x^T & ||\tilde{x}||_2^2
\end{bmatrix}
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u,v∼N(0,Λ)
u,v \sim \mathcal{N}(0,\Lambda)
Λ=[∣∣x∣∣22x⋅xTx⋅xT∣∣x~∣∣22]
\Lambda = \begin{bmatrix}
||x||_2^2 & x \cdot x^T \\
x \cdot x^T & ||\tilde{x}||_2^2
\end{bmatrix}
DUAL Activations
ϕˇ(ϵ)
\check{\phi}(\epsilon)
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u,v∼N(0,Λ)
u,v \sim \mathcal{N}(0,\Lambda)
Λ=[∣∣x∣∣22x⋅xTx⋅xT∣∣x~∣∣22]
\Lambda = \begin{bmatrix}
||x||_2^2 & x \cdot x^T \\
x \cdot x^T & ||\tilde{x}||_2^2
\end{bmatrix}
DUAL Activations
ϕˇ(ϵ)
\check{\phi}(\epsilon)
ϕˇ(ϵ)=E[ϕ(u)ϕ(v)]
\check{\phi}(\epsilon) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u,v∼N(0,Λ)
u,v \sim \mathcal{N}(0,\Lambda)
Λ=[1ϵϵ1]
\Lambda = \begin{bmatrix}
1 & \epsilon\\
\epsilon & 1
\end{bmatrix}
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
K(x,x~)=E[ϕ(u)ϕ(v)]
K(x,\tilde{x}) = \mathop{\mathbb{E}} \left [ \phi(u) \phi(v) \right ]
u,v∼N(0,Λ)
u,v \sim \mathcal{N}(0,\Lambda)
Λ=[∣∣x∣∣22x⋅xTx⋅xT∣∣x~∣∣22]
\Lambda = \begin{bmatrix}
||x||_2^2 & x \cdot x^T \\
x \cdot x^T & ||\tilde{x}||_2^2
\end{bmatrix}
ϕˇ(xx~T)
\check{\phi}(x \tilde{x}^T)
If data is normalized
NNGP
Neural Network Gaussian Process
B
B
A
A
ϕ.
\phi.
k→∞lim
\lim_{k\rightarrow \infty}
How to calculate the kernel matrix K?
f(x)=k1Aϕ(B⋅x)
f(x) = \frac{1}{\sqrt{k}} A \phi(B \cdot x)
ψ(x)=k1ϕ(B⋅x)
\psi(x) = \frac{1}{\sqrt{k}} \phi(B \cdot x)
K(x,x~)=k1i∑kϕ(Bi,:⋅x)⋅ϕ(Bi,:⋅x)
K(x,\tilde{x}) = \frac{1}{k} \sum_i^{k} \phi(B_{i,:} \cdot x ) \cdot \phi(B_{i,:} \cdot x)
wT
w^T
wT
w^T
K(x,x~)=Ew∼N(0,1)[ϕ(wTx)ϕ(wTx~)]
K(x,\tilde{x}) = \mathop{\mathbb{E}}_{w \sim \mathcal{N}(0,1)} \left [ \phi(w^T x) \phi(w^T \tilde{x}) \right ]
NTK
Neural Tangent Kernel
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
w=[A11⋯A1k∣B11⋯Bkd]
w = \begin{bmatrix}
A_{11} &\cdots & A_{1k} | B_{11} & \cdots B_{kd}
\end{bmatrix}
w∈Rk+kd
w \in \mathbb{R}^{k+kd}
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
w=[A11⋯A1k∣B11⋯Bkd]
w = \begin{bmatrix}
A_{11} &\cdots & A_{1k} | B_{11} & \cdots B_{kd}
\end{bmatrix}
"Linearize f around some initialization w"
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
w=[A11⋯A1k∣B11⋯Bkd]
w = \begin{bmatrix}
A_{11} &\cdots & A_{1k} | B_{11} & \cdots B_{kd}
\end{bmatrix}
"Linearize f around some initialization w"
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
w=[A11⋯A1k∣B11⋯Bkd]
w = \begin{bmatrix}
A_{11} &\cdots & A_{1k} | B_{11} & \cdots B_{kd}
\end{bmatrix}
"Linearize f around some initialization w"
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
∇fx=∂A11∂f∂A12∂f⋮∂Bkd∂f
\nabla f_x = \begin{bmatrix}
\frac{\partial f}{\partial A_{11}} \\
\frac{\partial f}{\partial A_{12}} \\
\vdots \\
\frac{\partial f}{\partial B_{kd}} \\
\end{bmatrix}
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
Unknown Weights
Can be 0
Then This is a linear model in w
L(w)=i=1∑N(yi−⟨w,∇fxi(w0)⟩)2
L(w) = \sum_{i=1}^N (y_i - \langle w , \nabla f_{x_i} (w^0)\rangle)^2
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
Unknown Weights
Can be 0
Then This is a linear model in w
L(w)=i=1∑N(yi−⟨w,∇fx(w0)⟩)2
L(w) = \sum_{i=1}^N (y_i - \langle w , \nabla f_x (w^0)\rangle)^2
This can be intepreted as mapping the data and solving a linear model around w0
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
L(w)=i=1∑N(yi−⟨w,∇fx(w0)⟩)2
L(w) = \sum_{i=1}^N (y_i - \langle w , \nabla f_x (w^0)\rangle)^2
ϕ:x→∇fx(w0)
\phi: x \rightarrow \nabla f_x (w^0)
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
L(w)=i=1∑N(yi−⟨w,∇fx(w0)⟩)2
L(w) = \sum_{i=1}^N (y_i - \langle w , \nabla f_x (w^0)\rangle)^2
ϕ:x→∇fx(w0)
\phi: x \rightarrow \nabla f_x (w^0)
K(x,x~)=⟨∇fx(w0),∇fx~(w0)⟩
K(x,\tilde{x}) = \langle \nabla f_x(w^0) , \nabla f_{\tilde{x}}(w^0) \rangle
NTK
f(x)=k1Aϕ(Bx)
f(x) = \frac{1}{\sqrt{k}} A \phi(Bx)
f(w;x)=k1Aϕ(Bx)
f(w; x) = \frac{1}{\sqrt{k}} A \phi(Bx)
fx(w)=fx(w0)+∇fx(w0)T(w−w0)
f_x(w) = f_x(w^0) + \nabla f_x(w^0)^T (w-w_0)
L(w)=i=1∑N(yi−⟨w,∇fx(w0)⟩)2
L(w) = \sum_{i=1}^N (y_i - \langle w , \nabla f_x (w^0)\rangle)^2
ϕ:x→∇fx(w0)
\phi: x \rightarrow \nabla f_x (w^0)
K(x,x~)=⟨∇fx(w0),∇fx~(w0)⟩
K(x,\tilde{x}) = \langle \nabla f_x(w^0) , \nabla f_{\tilde{x}}(w^0) \rangle
Neural Tangent Kernel
NTK
K(x,x~)=⟨∇fx(w0),∇fx~(w0)⟩
K(x,\tilde{x}) = \langle \nabla f_x(w^0) , \nabla f_{\tilde{x}}(w^0) \rangle
f(w;x)=Aϕ(Bx)=i∑kA1iϕ(Bi,:x)
f(w;x) = A \phi(Bx) = \sum_i^k A_{1i} \phi(B_{i,:} x)
∂A1i∂f=i∑ϕ(Bi,:x)
\frac{\partial f}{\partial A_{1i}} = \sum_i \phi(B_{i,:} x)
∂Bij∂f=i∑A1iϕ′(Bi,:x)xj
\frac{\partial f}{\partial B_{ij}} = \sum_i A_{1i} \phi'(B_{i,:} x) x_j
NTK
K(x,x~)=⟨∇fx(w0),∇fx~(w0)⟩
K(x,\tilde{x}) = \langle \nabla f_x(w^0) , \nabla f_{\tilde{x}}(w^0) \rangle
f(w;x)=Aϕ(Bx)=i∑kA1iϕ(Bi,:x)
f(w;x) = A \phi(Bx) = \sum_i^k A_{1i} \phi(B_{i,:} x)
∂A1i∂f=i∑ϕ(Bi,:x)
\frac{\partial f}{\partial A_{1i}} = \sum_i \phi(B_{i,:} x)
∂Bij∂f=i∑A1iϕ′(Bi,:x)xj
\frac{\partial f}{\partial B_{ij}} = \sum_i A_{1i} \phi'(B_{i,:} x) x_j
From Linear Systems to NNs Aadharsh Aadhithya A
deck
By Incredeble us
deck
- 106
