abdellah Chkifa
Some of my presentations
An introduction to gradient descent algorithms
Abdellah CHKIFA
abdellah.chkifa@um6p.ma
Outline
1. Gradient Descent
- Gradient Descent
- Linear regression
- Batch Gradient Descent
- Stochastic Gradient Descent
2. Back-propagation
- Logistic regression
- Soft-max regression
- Neural networks
- Deep neural networks
Gradient Descent: an intuitive algorithm
objective function (also called loss function or cost function etc.)
θ(k+1)=θ(k)−η∇f(θ(k))
\theta^{(k+1)} = \theta^{(k)} - \eta \nabla f (\theta^{(k)})
f
f
θ(0)
\theta^{(0)}
initial guess
minθ∈Ωf(θ)
min_{\theta \in \Omega} f(\theta)
Importance of learning rate
We consider GD applied to
f:x↦x2
f: x\mapsto x^2
with
x0=4
x_0=4
and rates
η=0.1
\eta=0.1
η=0.01
\eta=0.01
η=0.9
\eta=0.9
η=1.01
\eta=1.01
Gradient Descent: a convergence theorem
θ(k+1)=θ(k)−η∇f(θ(k))
\theta^{(k+1)} = \theta^{(k)} - \eta \nabla f (\theta^{(k)})
θ(0)
\theta^{(0)}
initial guess
f(tθ+(1−t)θ′)≤tf(θ)+(1−t)θ′,θ,θ′∈Rd,t∈[0,1]
f(t\theta + (1-t)\theta') \leq t f(\theta) + (1-t) \theta',\quad\quad \theta,\theta'\in{\mathbb R^d}, t\in [0,1]
f convex:
f L-smooth:
∥∇f(θ)−∇f(θ′)∥≤L∥θ−θ′∥,θ,θ′∈Rd.
\| \nabla f(\theta) - \nabla f(\theta')\| \leq L \|\theta-\theta'\|,\quad\quad \theta,\theta'\in{\mathbb R^d}.
Suppose that f is convex and L-smooth. The gradient descent algorithm with η < 1/L converge to θ* and yields the convergence rate
2
f:Rd→R
f: \R^d \to R
0≤f(θ(k))−f(θ∗)≤2ηk∥θ(0)−θ∗∥2
0 \leq f(\theta^{(k)}) - f(\theta^*) \leq \frac {\|\theta^{(0)} - \theta^*\|^2}{2\eta k}
Theorem
Gradient Descent: a convergence theorems
θ(k+1)=θ(k)−η∇f(θ(k))
\theta^{(k+1)} = \theta^{(k)} - \eta \nabla f (\theta^{(k)})
θ(0)
\theta^{(0)}
initial guess
f μ-strongly convex:
f:Rd→R
f: \R^d \to R
f(θ′)≥f(θ)+⟨∇f(θ),θ′−θ⟩+2μ∥θ′−θ∥2,θ,θ′∈Rd.
f(\theta') \geq f(\theta) +
⟨
\nabla f(\theta), \theta'-\theta
⟩
+ \frac \mu 2 \|\theta'-\theta\|^2
,\quad \quad \theta,\theta'\in{\mathbb R^d}.
suppose that f is μ-strongly convex and L-smooth. The gradient descent algorithm with η < 1/L converge to θ* with
∥θ(k)−θ∗∥≤(1−ημ)k∥θ(0)−θ∗∥
\| \theta^{(k)} - \theta^* \| \leq (1-\eta \mu)^k \|\theta^{(0)} - \theta^* \|
Theorem
μ-strong convexity can be weakened to Polyak-Lojasiewicz Condition 🔗
- Easy to implement.
- Each iteration is (relatively!) cheap, computation of one gradient.
- Can be very fast for smooth objective functions, as shown.
- The coefficient L may not be accurately estimated.
- => Choice of small learning rate.
- Concrete objective functions are usually not strongly convex nor convex.
- GD does not handle non-differentiable functions (biggest downside).
Pros and Cons
Reference
https://www.stat.cmu.edu/~ryantibs/convexopt/
An introduction to gradient descent algorithms Abdellah CHKIFA abdellah.chkifa@um6p.ma
IndabaX_Morocco_2019
By abdellah Chkifa
IndabaX_Morocco_2019
- 1,483
