Accounting for Real World Phenomena in Machine Learning and Mechanism Design
Nicholas Bishop
Decision-Making in Theory
(\textcolor{blue}{\mathbf{x}}, \textcolor{red}{y})
Real World Complications
(\textcolor{blue}{\mathbf{x}}, \textcolor{red}{y})
Insurance
$500
$470
Insurance
$450
$470
Insurance
Inspection doesn't scale!
Real World Complications
(\textcolor{blue}{\mathbf{x}}, \textcolor{red}{y})
Resource Deployment
Resource Deployment
Resource Deployment
Resource Deployment
Resource Deployment
We must account for resource availability!
Goals
Consider agent incentives when making decisions
Account for real world constraints on decision
making
Contributions
Stackelberg Prediction Games for Linear Regression (Chapter 3)
Adversarial Blocking Bandits (Chapter 4)
Sequential Blocked Matching (Chapter 5)
Strategic Linear Regression
\{(\textcolor{blue}{\mathbf{x}_{i}}, \textcolor{red}{y_{i}}, \textcolor{purple}{z_{i}})\}^{m}_{i=1}
(\textcolor{blue}{\mathbf{x}}, \textcolor{red}{y}, \textcolor{purple}{z_{i}})
At training time:
At test time:
(\textcolor{blue}{\tilde{\mathbf{x}}}, \textcolor{red}{y}, \textcolor{purple}{z_{i}})
Strategic Linear Regression
\{(\textcolor{blue}{\mathbf{x}_{i}}, \textcolor{red}{y_{i}}, \textcolor{purple}{z_{i}})\}^{m}_{i=1}
(\textcolor{blue}{\mathbf{x}}, \textcolor{red}{y}, \textcolor{purple}{z})
At training time:
At test time:
(\textcolor{blue}{\tilde{\mathbf{x}}}, \textcolor{red}{y}, \textcolor{purple}{z})
Target of the data provider
Choosing a Predictor
Idea: Simulate agent behaviour using training data!
\min_{\mathbf{w}}\sum^{m}_{i=1}(\mathbf{w}^{\top}\textcolor{blue}{\mathbf{x}^{\star}} - \textcolor{red}{y_{i}})^{2}
\text{s.t. } \textcolor{blue}{\mathbf{x}_{i}^{\star}} = \text{arg}\min_{\textcolor{blue}{\tilde{\mathbf{x}}_{i}}}
(\mathbf{w}^{\top}\textcolor{blue}{\tilde{\mathbf{x}}} - \textcolor{purple}{z_{i}})^{2} + \gamma\|\textcolor{blue}{\tilde{\mathbf{x}}_{i}}
- \textcolor{blue}{\mathbf{x}_{i}}\|^{2}_{2} \quad \forall i
Learner's loss
Agent's loss
Agent's manipulation cost
Key Questions
Can we solve such an optimisation problem?
Does the solution generalise well?
Key Questions
Can we solve such an optimisation problem?
Does the solution generalise well?
Optimisation
Reformulate the problem as a fractional program
\text{arg}\min_{\mathbf{w}}\left\|\frac{\frac{1}{\gamma}\textcolor{purple}{\mathbf{z}}\|\mathbf{w}\|^{2} + \textcolor{blue}{X}\mathbf{w}}{
(1 + \frac{1}{\gamma}\|\mathbf{w}\|^{2})^{2}} - \textcolor{red}{\mathbf{y}}\right\|^{2}
Substitute outÂ
\|\mathbf{w}\|^{2}
\arg\min_{\mathbf{w}, \alpha}\left\|\frac{\frac{\alpha}{\gamma}\textcolor{purple}{\mathbf{z}} + \textcolor{blue}{X}\mathbf{w}}{
(1 + \frac{\alpha}{\gamma})^{2}} - \textcolor{red}{\mathbf{y}}\right\|^{2}
\text{s.t. } \alpha = \|\mathbf{w}\|^{2}
Optimisation
Use fractional programming to rewrite this problem as single parameter root finding problem.
\arg\min_{\mathbf{w}, \alpha}\left\|\frac{\frac{\alpha}{\gamma}\textcolor{purple}{\mathbf{z}} + \textcolor{blue}{X}\mathbf{w}}{
(1 + \frac{\alpha}{\gamma})^{2}} - \textcolor{red}{\mathbf{y}}\right\|^{2}
\text{s.t. } \alpha = \|\mathbf{w}\|^{2}
F(q) = \text{arg}\min_{\mathbf{w}, \alpha}\|\frac{\alpha}{\gamma}\textcolor{purple}{\mathbf{z}} + \textcolor{blue}{X}\mathbf{w} -
\textcolor{red}{\mathbf{y}}
- \frac{\alpha}{\gamma}\textcolor{red}{\mathbf{y}}\|^{2}
- q(1 + \frac{\alpha}{\gamma})^{2}
\text{s.t. } \alpha = \|\mathbf{w}\|^{2}
= 0
Idea: Use bisection search to find a root!
Optimisation
Problem: How do we evaluate       Â
F(q)\text{?}
\text{s.t. }\begin{bmatrix}
A + \lambda B & \mathbf{a} + \lambda \mathbf{b} \\
\mathbf{a}^{\top} + \lambda \mathbf{b}^{\top} & c - \tau
\end{bmatrix}
\max_{\tau, \lambda} \:\tau
Solution: Convert to an SDP!       Â
Generalisation
h_{\mathbf{w}}(\textcolor{blue}{\mathbf{x}}, \textcolor{purple}{z}) = \frac{\mathbf{w}^{\top}\textcolor{blue}{\mathbf{x}}
+ \frac{1}{\gamma}\|\mathbf{w}\|^{2}\textcolor{purple}{z}}{1 + \frac{1}{\gamma}\|\mathbf{w}\|^{2}}
Consider what a linear function predicts after manipulation:
Each of these functions are linear and have bounded norm!
Hence we can bound the Rademacher complexity of the resulting hypothesis class!
Blocking Bandits
\textcolor{blue}{\mu_{1}}
\textcolor{blue}{\mu_{2}}
\textcolor{blue}{\mu_{3}}
\textcolor{red}{D_{1}}
\textcolor{red}{D_{2}}
\textcolor{red}{D_{3}}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
\textcolor{green}{1.5}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
\textcolor{green}{0.9}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
Blocking Bandits
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
Rewards and Delays in the Real World
\textcolor{blue}{1}
\textcolor{blue}{5}
\textcolor{blue}{8}
\textcolor{red}{1}
\textcolor{red}{2}
\textcolor{red}{2}
Rewards and Delays in the Real World
\textcolor{blue}{10}
\textcolor{blue}{11}
\textcolor{blue}{2}
\textcolor{red}{3}
\textcolor{red}{5}
\textcolor{red}{3}
Rewards and Delays in the Real World
\textcolor{blue}{10}
\textcolor{blue}{11}
\textcolor{blue}{2}
\textcolor{red}{3}
\textcolor{red}{5}
\textcolor{red}{3}
Rewards and Delays in the Real World
\textcolor{blue}{2}
\textcolor{blue}{20}
\textcolor{blue}{10}
\textcolor{red}{6}
\textcolor{red}{7}
\textcolor{red}{6}
Adversarial Blocking Bandits
Rewards vary adversarially in accordance with a path variation budget
Blocking durations are free to vary arbitrarily, but are bounded above.
\sum^{T-1}_{t=1}\sum^{K}_{k=1}|X^{k}_{t+1} - X^{k}_{t}| \leq B_{T}
D^{k}_{t} \leq \tilde{D} \quad \forall k , t
Full Information Setting
Consider a greedy algorithm which pulls the arm with highest reward
Using a knapsack-style proof, we obtain the following regret guarantee
\text{arg}\max_{k \in A_{t}}X^{k}_{t}
\left(1 + \tilde{D}\right)^{-1}\left(1 - \frac{\tilde{D}B_{T}}{r(\pi^{\star})}\right)
Bandit Setting
0
T
Split the time horizon into blocks
Bandit Setting
0
T
\Delta_{T}
Split the time horizon into blocks
Bandit Setting
0
T
Split the time horizon into blocks
Consider one such block
At the start of the block, play each arm once, and store the rewards observed. Then pull no arms until all arms are available.
Bandit Setting
0
T
Split the time horizon into blocks
Consider one such block
Then play greedily, using the rewards received in the first phase as a proxy for the real reward
Bandit Setting
0
T
Split the time horizon into blocks
Consider one such block
Pull no arms at the end of the block so all arms will be available at the beginning of the next block.
Bandit Setting
0
T
Split the time horizon into blocks
Consider one such block
Pull no arms at the end of the block so all arms will be available at the beginning of the next block.
Bandit Setting
By appropriately choosing block length, we can obtain the following regret bound:
\mathcal{O}\left(\sqrt{T(2\tilde{D} + K)B_{T}}\right)
Problem: We need to know the variation budget to set the block length!
Solution: Run EXP3 as a meta-bandit algorithm to learn the correct block length!
Bandit Setting
Maintain a list of possible budgets and split the time horizon into blocks
0
T
Bandit Setting
Maintain a list of possible budgets and split the time horizon into blocks
0
T
H
Bandit Setting
Maintain a list of possible budgets and split the time horizon into blocks
0
T
Consider one such block
Bandit Setting
Maintain a list of possible budgets and split the time horizon into epochs
0
T
Consider one such epoch
Sample a budget and thus an associated block length and play the previous algorithm within the epoch
Bandit Setting
Maintain a list of possible budgets and split the time horizon into epochs
0
T
Consider one such epoch
Sample a budget and thus an associated block length and play the previous algorithm within the epoch
Bandit Setting
Maintain a list of possible budgets and split the time horizon into epochs
0
T
Consider one such epoch
At the end of the epoch, update the sampling probability of the chosen budget according to EXP3.
Bandit Setting
Maintain a list of possible budgets and split the time horizon into epochs
0
T
Consider one such epoch
Repeat this process with the next epoch.
Sequential Blocked Matching
2
2
1
1
2
1
1
1
1
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Sequential Blocked Matching
2
1
1
1
1
2
1
1
2
2
1
1
Requirements
Resistance to strategic manipulation induced by blocking - bound the incentive ratio.
Achieve high social welfare - minimise the distortion.
Repeated RSD
Generalise RSD, by allowing each agent to choose its allocation for the entire time horizon greedily.
0
T
Repeated RSD
Generalise RSD, by allowing each agent to choose its allocation for the entire time horizon greedily.
0
T
Repeated RSD
Generalise RSD, by allowing each agent to choose its allocation for the entire time horizon greedily.
0
T
Repeated RSD
Generalise RSD, by allowing each agent to choose its allocation for the entire time horizon greedily.
0
T
Repeated RSD
Generalise RSD, by allowing each agent to choose its allocation for the entire time horizon greedily.
0
T
Repeated RSD
Repeated RSD is asymptotically optimal in terms of distortion.
Repeated RSD can be derandomized to yield a deterministic algorithm which is also asymptotically optimal!Â
Repeated RSD also has bounded incentive ratio!
Bandit Matching
Bandit Matching
Bandit Matching
i
j
m_{ij} \sim \mu_{ij}
Bandit Matching
i
j
m_{ij} \sim \mu_{ij}
\hat{\mu}_{i} = \frac{\sum^{T}_{t=1}\mathbf{1}[m_{i}(t) = j]m_{i}(t)}{\sum^{T}_{t=1}\mathbf{1}[m_{i}(t) = j]}
(mean-based)
Bandit RRSD
Idea: Extend RRSD to bandit setting with explore-then-commit framework!
0
T
Bandit RRSD
Idea: Extend RRSD to bandit setting with explore-then-commit framework!
0
T
In the exploration phase, assign each agent each service a fixed number of times
Bandit RRSD
Idea: Extend RRSD to bandit setting with explore-then-commit framework!
0
T
In the exploration phase, assign each agent each service a fixed number of times
Wait until all arms are available
Bandit RRSD
Idea: Extend RRSD to bandit setting with explore-then-commit framework!
0
T
In the exploration phase, assign each agent each service a fixed number of times
Wait until all arms are available
In the exploitation phase, play RRSD, using the last preference submission of each agent
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Bandit RRSD
0
T
2
2
1
1
1
2
1
1
2
1
1
1
8
1
9
7
10
3
9
4
1
5
7
0
Viva Voce
By nickbishop
