Real-world decision making through MABs and multi-level reinforcement learning on agent-based models

Harshavardhan Kamarthi

Presented at Data Seminar (Oct 8)

AI Economist

  • Learn optimal tax policies
    • Hard to experiment in real-world
    • Lucas critique
  • Fine-grained modeling of behavior
    • Ad-hoc, homogenous behavior rules
  • Flexible objectives and free parameters
    • easy to add or remove policy, action constraints
  • Learn directly from observations
    • Use experience from agents and planners to optimize
    • calibrate (macro) parameters from real-world data

Zheng et al

AI Economist for Tax Policies

  • Agent-based model: Gather-Build-Trade
  • Each agent maximizes its own utility by
    • Gathering resources
    • Building houses
    • Trading resources and houses for coins
  • High-level social planner
    • Levies tax on income
    • Redistributes the collected tax to everyone equally

Environment

  • 25 x 25 grid world
  • Each grid
    • Source cell: can spawn new resources stochastically
    • House cell: occupied by owner only
    • Water cell: can not be occupied
    • Other cells
  • Building: need one unit of stone and wood to build a house
  • Trading:
    • Continuous double auction
    • Buy and sell resources
  • Bracketed tax rate is levied for income earned as determined by planner

Agent State

  • Skills (Fixed)
    • Gathering skill (0-3)
    • Building skill (0-3)
  • Resources
    • Wood, stone
    • Coins
  • Trades
    • Current ask bids (up to 5)
    • Current buy bids (up to 5)
  • Location and surroundings (10x10 neighborhood)
  • Current tax rate

Agent Actions

  • Move actions (UDLR, No-op)
  • Build action (Y/N)
    • Get coins = (1+Building skill)
  • Gather action (if in source cell)
    • If available get 1 + (Gathering skill) resources (stochastic)
  • Trade
    • Add an ask/buy bid for any of the resources
    • Can add almost 5 bids per resources

Planner State

  • The full map of all agents' locations
  • Current tax rate
  • All bids in market
  • Resources and coins of all agents

Planner Action

  • Tax rate for each of the 7 tax brackets

Model

  • Tax rate for each of the 7 tax brackets

Agent Reward

  • Utility
    • Increases with number of coins
    • decreases with labor (moving, gathering, trading)
u_i(x_i,l_i) = crra(x_i) - l_i,\ crra(x) = \frac{x_i^{1-\eta - 1}}{1-\eta}
  • Consider decreasing marginal return over coins gathered
  • Goal: maximize marginal utility at each time step
E[\sum_{t=1}^T (\gamma_t u_{i,t} - u_{i, t-1}) + u_{i,0}]

Planner Reward

  • Tradeoff between equality and productivity
  • Equality: Gini-index over all post-tax income
  • Productivity: total wealth in economy
  • Reward:
swf_t = eq_t \times prod_t
E[\sum_{t=1}^T (\gamma_t swf_{t} - swf_{t-1}) + swf_{0}]
  • Goal: maximize marginal social welfare

RL Training

  • Two level training
  • Hard:
    • Reward function of agents depends on action of the planner
    • Reward function of planner depends of agent policies
    • MDPs are unstable
  • Tricks: Pre-train agents, Curriculum learning, Entropy regularization
  • Pre-train agents:
    • Pre train agent networks with random fixed tax policies before joint training
  • Curriculum learning: slowly increase variation in tax policies
    • Slowly increase the maximum tax rate for planner
  • Entropy regularization: improve stochasticity and exploration

Baselines

  • Agents' initial position - uniformly at random
  • Build, gather skills - Pareto distribution
  • Tax rate changes every 20 time steps

Initialization

  • No Taxes! (Free market)
  • US federal rate (Make brackets s.t USD rates proportional to coins)
  • Saez tax policy (optimal with homogenous assumptions on tax elasticity)

Results

Results

  • AI Econ. improves on the best baseline by 16%
  • 47% better equality than the free market at cost of only 11% decrease in productivity
  • Between progressive and regressive tax structure

Effect of planner on agent behavior

  • Regressive Saez policy makes builders trade less, making them collect resources on their own
  • Trading helps redistribute wealth (low tax on gatherers + more time to build)
  • Gaming the system: initially as planner is converging to optimal policy, agents also alternate total income earned to reduce taxes

What about human participants?

  • Used human participants instead of agents to play the game.
  • Incentives proportional total income earned at end of the game.

Optimizing Public Health and economy during the pandemic

  • Analyze and learn policies at Federal and State level
  • Federal (planner):
    •  Provide subsidies to states
    • Optimize social welfare
  • State: Maximize state level utilities

Optimizing Public Health and economy during the pandemic

Optimizing Public Health and economy during the pandemic

State Level Epidemic Model

  • uses a variant of SIR model that accounts for vaccination
  • Transmission rate  is location and time specific based on stringency level

 

  • Vaccination rate proportional to population
\beta_{i,t} = \beta_i^{\pi} \pi_{i,t-d} + \beta_0
\Delta V_i = v_i n_i

State Level Econ. Model

  • Unemployment proportional to stringency level
U_{i,t} = \text{softplus}\left( \sum_{k=1}^K w_{i,k} \tilde{U}_{i,t}^k \right) + U_i^0
\tilde{U}_{i,t}^k = \sum_{t'=t-L}^t e^{(t'-t)/\lambda_k} \Delta\pi_{i,t'}
  • Economic output depends on available workers and federal subsidies
\omega_{i,t} = c(n_i - D_{i,t} - \eta I_{i,t}) - U_{i,t}\\ P_{i,t} = k\omega_{i,t} + T_{i,t}

Rewards

  • Condiser health index and economic productivity
r_{i,t} = \alpha_i \Delta H_i + (1-\alpha_i) \Delta E_i\\ \Delta H_{i,t} = - \Delta D_{i,t}\\ \Delta E_{i,t} = crra\left( \frac{P_{i,t}}{P_i^0}\right)
  • At federal level aggregate state-level productivity except penalize for subsidies
P_{f,t} = \sum_{i} P_{i,t} - c T_{i,t}

Action and policy parametrization

  • Log-linear policies.
  • Interpretability over weights (same for all states) and bias (each state)
  • Discretize to 10 levels of stringency
  • 20 levels of subsidies
  • Actual subsidy proportional to population of state

Calibration

Datasets

  • Covid-19 Govt. policy Tracker: Stringency
  • Covid-19 Money Tracker -  subsidies
  • JHU death estimates - Deaths
  • Daily unemployment - BLS

parameters to calibrate

  • Death rate, vaccination rate(state), recovery rate
  • Baseline params: infection, unemployment, productivity
  • State level 
\alpha_i

Results

  • Simulation from March 22 2020 to April 30 2021
  • Calibration fits well with real world
  • AI imposes higher stringency
  • But unemployment is usually larger

Results

  • 4.7% higher federal welfare (reward)
  • Better balance between unemployment and deaths (?)
  • Much lower subsidies ($35B vs $630B)

Results

  • Changing priority factor to focus more on health, AI econ does even better.

Interpretability

  • Effect on individual features on stringency
  • State-wise bias on stringency

RL for border testing policies

  • Leverage demographic features from travelers to provide estimates of infection prevalence
  • Use prevalence estimates to better sample travelers for testing
  • Deployed in real world - 40 Ports of entry in Greece

Bastani et al

Traveller data

  • Used anonymized categorical features: Country, Region, Age, Gender
  • Used Lasso regularize on past data to predict positive case to select important features
  • This determines classes of travellers for test allocation

Problem

  • Allocate tests         to each type for day t
  • Subject to:
    • Total budget for tests
  • Assume          is probability of bing positive for class k
T_k(t)
R_k(t)
max \sum_t \sum_k T_k(t) R_k(t)

But:

  • Prevalence probability is unknown
  • May Change
  • We need to decide since day 0
    • Allocate higher tests for those whose estimates are high
    • Allocate tests to classes for which we have low certainty
    • Exploit vs Explore Dilemma

Estimation

  • Use empirical Bayes to estimate new prevalence based on past 14 days cases
  • Use a Beta prior: the prior parameters for all classes of same country is same (weight sharing)

Allocation

  • Recall: Explore vs Exploit dilemma
  • Use Multi Arm Bandits
  • Specifically used modified version of Optimistic Gittins Index
    • Good guarantees for batched decision making
    • Similar motivation to UCB methods (Optimism in face of uncertainty)

Results

  • 54.7% better than random allocation (73% during Oct peak)
  • Using Cases, deaths, positive rates to allocate also underperforms significantly
  • Using estimated prevalence probability to grey-list countries prevented 6.7% less infected people from entering
Made with Slides.com