ViTAL

Vision-Based Terrain-Aware Locomotion

for Legged Robots

Shamel Fahmi

November, 2021

Research Statement

Let legged robots take decisions based on the surrounding terrain using their perception.

decisions

terrain

perception

Let legged robots take decisions based on the surrounding terrain using their perception.

Passive Whole-Body Control (pWBC)

STANCE: Locomotion Adaption over Soft Terrain

On State Estimation for Legged Robots over Soft Terrain

ViTAL: Vision-based Terrain-Aware Locomotion

Perception

  • Control

  • State Estimation

  • Planning

Terrain

  • Proprioception

  • Vision

Decisions

  • Physical Properties

    • Friction

    • Impedance

  • Geometry

Using Optimization and Machine Learning

Locomotion Planning

Two ways to deal with locomotion planning:

1. Coupled Planning:

  • Plan the feet & base states simultaneously.

2. Decoupled planning:

  • Separate the plan into feet & base plans.

  • Planning the feet motion ⇒ foothold selection

  • Planning the body motion ⇒ pose adaptation

  • Planning the feet motion ⇒ foothold selection

Vision-Based Foothold Adaptation (VFA)

Esteban 2020

  • Better Evaluation Criteria

  • Improved CNNs

Villarreal 2019

  • Better Evaluation Criteria

  • Self-supervised via CNNs

Barasuol 2015

  • Simple Evaluation Criteria

  • Expert demonstrations

Foothold Selection

V. Barasuol et al., Reactive Trotting with Foot Placement Corrections through Visual Pattern Classification, IROS, 2015.
O. Villarreal et al., Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs, RA-L, 2019.
D. Esteban et al., On the Influence of Body Velocity in Foothold Adaptations for Dynamic Legged Locomotion via CNNs, CLAWAR, 2020.

Goal: Select footholds based on the terrain information and the robot's capabilities. 

  • Despite the work done on vision-based strategies for foothold selection, pose adaptation strategies lag behind.

 

 

  • Pose adaptation is important because to reach a certain foothold, the robot's pose has to move simultaneously with the legs.

What is the problem?

Pose Adaptation

  • The goal is to find the optimal robot's pose that can:

    • Maximize the reachablity of the selected footholds.

    • Avoid terrain collision and maintain static stability.

 

  • An pose optimizer to deal with these objectives.

M. Kalakrishnan et al., Learning, planning, and control for quadruped locomotion over challenging terrain, IJRR 2011.
P. Fankhauser et al., Robust Rough-Terrain Locomotion with a Quadrupedal Robot, IROS, 2018.

Kalakrishnan 2011

Fankhauser 2018

Goal: Adapt the robot's pose based on the terrain information and the robot's capabilities. 

The Problem with Pose Adaptation

  • The problem with current pose adaptation strategies is that they focus on finding one optimal solution based on given selected footholds.

 

  • First, the footholds are selected.

  • Then, the pose is optimized based on that.

 

  • What if the given footholds not reached (if the robot gets disturbed)?

 

  • The robot may end up in a state with no reachable safe footholds. 

  • Propose a different paradigm for pose adaptation.

 

  • Instead of finding body poses that are optimal w.r.t given footholds,

 

  • we find body poses that will maximize the chances of the robot to reach safe footholds.

 

  • Put the robot in a state that if it gets disturbed, it remains around a set of solutions that are still safe & reachable.

How to Solve This Problem?

Vision-Based Terrain Aware Locomotion (ViTAL)

  • ViTAL is an online vision-based locomotion planning strategy.

 

  • ViTAL selects footholds based on the robot's skills

 

  • Also, find the robot pose that maximizes the chances of success in reaching these footholds.

 

  • The notion of safety & success emerges from skills that characterize the robot's capabilities.

Vision-Based Foothold Adaptation (VFA)

Vision-Based Pose Adaptation (VPA)

Foothold Evaluation Criteria (FEC)

What the robot is capable of doing?

  • Avoid edges, corners, gaps, etc.
  • Remain with the workspace of the legs.
  • Avoid colliding with the terrain.

Foothold Evaluation Criteria (FEC)

The FEC is a way to evaluate footholds given a heightmap, the leg's hip height, & other robot states.

When the criteria are evaluated, a boolean matrix is given as an output.

Each pixel in the heightmap represents the terrain height corresponding to the pixel's location in the terrain map.

The heightmap is a 2D representation of the terrain.

Each pixel is a candidate foothold for the robot.

The boolean matrix characterizes which pixel (location) in the heightmap is feasible (safe) or not based on the robot's capabilities (the criteria). 

Foothold Evaluation Criteria (FEC)

Rejects footholds that cause foot trajectory collision.

- Foot Trajectory Collision (FTC)

Rejects footholds that are outside the workspace during the entire gait phase.

-  Kinematic Feasibility (KF)

Rejects footholds that cause leg collision with the terrain.

-  Leg Collision (LC)

-  Terrain Roughness (TR)

Rejects footholds that are near holes, spikes, edges, and corners.

Vision-Based Foothold Adaptation (VFA)

From the VFA to the VPA

VFA

VPA

Vision-Based Pose Adaptation (VPA)

First, we extract heightmaps that are centered around the projection of the leg hip location.

Vision-Based Pose Adaptation (VPA)

Then, the pose evaluation stage will evaluate the criteria for all the samples of the hip heights. 

The pose evaluation stage will output samples of number of safe footholds.

Vision-Based Pose Adaptation (VPA)

The output of the function approximation is

\hat{\mathcal{F}}_l

These samples are then used to approximate a continuous function of the number of safe footholds at the function approximation stage. 

Vision-Based Pose Adaptation (VPA)

Finally, the pose optimization stage finds the optimal pose that maximizes            for all of the legs

\hat{\mathcal{F}}_l

Pose Optimization

Find the optimal robot's trunk pose that maximizes the number of safe footholds for all the legs

Height

Roll

Pitch

Pose Optimization

Results: Detailed Explanation

hind hips rising up

hind hips lowering down

front hips rising up

front hips lowering down

Baseline: Terrain-Based Reference (TBR)

O. Villarreal et al., MPC-based Controller with Terrain Insight for Dynamic Legged Locomotion, ICRA, 2020.

Results: Comparison with the baseline

Results: Comparison with another approach

Results: Performance with Different Velocities

Conclusions: ViTAL

  • We presented ViTAL, a decoupled visual planning strategy

 

  • ViTAL consisted of :

  • VFA for foothold selection

  • VPA for pose adaptation

    • Introduced a different paradigm in pose adaptation

    • Finds the body poses that maximize the chances of the legs to succeed in reaching safe footholds

  • FEC (notion of safety) that characterizes the robot's capabilities

Conclusion: ViTAL

  • ViTAL allows HyQ and HyQReal to traverse a wide variety of terrains under various forward velocities.

    • These terrains included stairs, gaps, etc.

    • Forward velocities varied from 0.2m/s to 0.75 m/s.

 

  • We showed that the VPA resulted in body poses that are aware of the terrain, and of what the robot and its legs can do.

 

  • The VPA puts the robot in a pose that provides the feet with higher number of safe footholds which allows the robot to succeed unlike the baseline that fails.

Limitations of ViTAL

  • State Estimation and Mapping

 

  • Tracking the Reference Motion

Future Directions

  • Optimization-based Skills (FEC)

 

  • Commanding forward velocities

 

  • Confined Spaces

 

  • Dynamically feasible ViTAL

 

  • FEC analysis on Mini Cheetah Vision (v2)?

On Terrain Aware Locomotion for 

Legged Robots

- SHAMEL FAHMI -

Made with Slides.com