Book club, 30.07.2021
Claudia Merger, Alexandre René
Ideas, examples,
context
Proofs, definitions,
"clean" textbook
Chapters 1-3
todays topics
Learning machine:
Given samples
fit some function
to minimize some risk
If the learning machine obeys ERM inductive principle for any given set of observations, call it a learning process.
Learning machine:
Given samples
fit some function
to minimize some risk
Only minimize the empirical risk?
Learning machine:
Given samples
fit some function
to minimize some risk
Only minimize the empirical risk?
Task: Learn Data p.d.f.
"Solution": Mixture with one component per data point:
What we really want to minimize:
according to the true p.d.f.
What we really want to minimize:
according to the true p.d.f.
but we don't have
How do we know when we're close?
For each sample of size pick
such that
For each sample of size pick
such that
the learning process is consistent iff
and
For each sample of size pick
such that
the learning process is consistent iff
and
choose the right
get correct risk
Task: Learn Data p.d.f.
"Solution": Mixture with one component per data point:
Task: Learn Data p.d.f.
"Solution": Mixture with one component per data point:
for some
for some
for some
trivially
trivial consistency
for some
trivially
for some
trivially
trivial consistency
Non-trivial consistency
consistency on any subset
Non-trivial consistency
consistency on any subset
in the following consistent = nontrivially consistent
proof: Statistical Learning Theory, p. 89-92
What are the necessary and sufficient conditions?
For some
For some
For some
scaling with ?
For some
scaling with ?
Which properties must have?
K. Popper: A theory is scientific, if it is falsifiable.
K. Popper: A theory is scientific, if it is falsifiable.
Example: " What goes up, must come down." - Falsifiable?
K. Popper: A theory is scientific, if it is falsifiable.
Example: " What goes up, must come down." - Falsifiable?
Example: "Whatever will be, will be." - Falsifiable?
K. Popper: A theory is scientific, if it is falsifiable.
Example: " What goes up, must come down." - Falsifiable?
Example: "Whatever will be, will be." - Falsifiable?
Task: Learn Data p.d.f.
"Solution": Mixture with one component per data point:
Task: Learn Data p.d.f.
"Solution": Mixture with one component per data point:
For consistency, the set of functions
may not be too "flexible".
Given a set of functions
Given a set of functions
and samples
Given a set of functions
and samples
construct set of vectors
Given a set of functions
and samples
construct set of vectors
minimal
Minimal number
of vectors such that all
have at maximum distance .
Given a set of functions
and samples
construct set of vectors
minimal
Minimal number
of vectors such that all
have at maximum distance .
Expectation of the diversity of the set of functions on a sample of size .
sufficient
Example:
indicator functions
input space
input space
Example:
Example:
Example:
worst case: for some
Example:
worst case: for some
Not falsifiable + Inconsistent
Problems
input space
For some
scaling with ?
the learning process is consistent iff
and
choose the right
get correct risk
All we need to know about learning?
All we need to know about learning?
No
All we need to know about learning?
No
if small
tradeoff between empirical risk minimization and VC dimension
All we need to know about learning?
No
if small
tradeoff between empirical risk minimization and VC dimension
add regularization