Empowering Physically Assistive Robots with Multimodal Active Learning

Ethan K. Gordon

PhD 2023, University of Washington; Postdoc, University of Pennsylvania

Work supported by: NSF (IIS; DMS; NRI; CHS; GRFP; FRR), NIH, Amazon, and RAI

ethan@ethankgordon.com

PREVIOUS WORK: ROBOT-ASSISTED FEEDING

Computing \(g=\nabla \log p(m_t | x_T)\)

ONGOING WORK: ACTIVE TACTILE EXPLORATION

Next Step

Translate to contact-rich physical activities of daily living (ADLs) with visual occlusions.

Dressing, Bathing, etc.

FUTURE DIRECTIONS: SAFE MULTI-FUNCTION ASSISTIVE SYSTEMS

RESEARCH AGENDA

Empowerment is the promise of Physically Assistive Robots (PARs)

If I can have a robot [working with me], it would be me feeding me, and that would be a huge deal.

-Community Researcher with a Spinal Cord Injury

• Goal: Work with community members to develop the technologies necessary to achieve that promise.
• User Discussions inform New Technologies support System Design
• Methods:
  • Community-Based Participatory Research: Involve stakeholders in system co-design, pre-build observations, and user studies
  • Active Learning:  Users wanted robots that can explore for information and exploit that understanding
  • Reasoning Through Contact: Leverage haptic sensing and physical contact models to enhance safety and data efficiency

Leverage expert heuristics and data-driven priors for policy-space reduction.

At deployment time: share post hoc haptic info with a vision-based contextual bandit to learn the best policy for new, never-before-seen food items.

Reached user-defined 80% success threshold within 8 Attempts

Emergent Policies:

Wiggling

Titing

High-Pressure

Scooping

Bring on community researchers (CR) as co-designers to create a system robust enough to be deployed in public and the home.

CR: Tyler

CR: Jonathan

• Portability: powered from wheelchair battery, no internet
• Safety: Watchdog monitoring force sensor at 120Hz and stopping for any anomaly
• Customization: Enable full control of the state machine within an accessible app

Tractably learn the shape and pose of arbitrary, dynamic rigid bodies using only tactile data.

Leverage the mathematical structure of contact dynamics to learn and quantify information through time and take actions that maximize expected info gain (EIG).

Safer to play pessimistically w.r.t. model parameters

Safe Active Exploration

As et al, "ActSafe...", ICLR 2025

Beneficial to play optimistically w.r.t. loss

vs.

Possible Solution, which elements are:

Safety-Critical

(zero user error tolerance)

vs.

Performance-Critical

(higher user error tolerance)

Multi-ADL PARs

1. Single system or connected ensemble?
2. Info sharing between tasks?
3. Longitudinal studies: in-home 24/7 for weeks or months?

Hello Robot

Kapusta 2019

Kinova