live.shamelfahmi.com

The Dynamic Legged Systems (DLS) Lab

• Part of the Italian Institute of Technology, Genoa, Italy

• 14 years of expertise in legged systems research

• Robot Design, Locomotion, (Hydraulic) Control, Perception, etc.

• Focus on heavy duty robots over rough terrain

The Dynamic Legged Systems (DLS) Lab

HyQ: Early Work (2008-2013)

• The first steps of HyQ

• Mainly open loop with pre-generated trajectories 

C. Semini et al., Design of HyQ - a Hydraulically and Electrically Actuated Quadruped Robot JSCE, 2011.

HyQ: Early Work (2008-2013)

• Focus on low level torque control (joint, leg and torso)

• Blind locomotion

V. Barasuol et al.,A Reactive Controller Framework for Quadrupedal Locomotion on Challenging Terrain, ICRA, 2013.
C. Semini et al., Towards versatile legged robots through active impedance control, IJRR, 2015.
T. Boaventura et al., Model-Based Hydraulic Impedance Control for Dynamic Robots Robotics, T-RO, 2015.

HyQ2Max (2014-2016)

• 2nd generation of the HyQ flagship robots

• Challenges due to HyQ's workspace

• HyQ2Max's workspace allows it to self-right

C. Semini et al, Design of the Hydraulically-Actuated, Torque-Controlled Quadruped Robot HyQ2Max, T-Mech, 2016.

HyQ2Max (2014-2016)

HyQReal (2019-)

• What did HyQ2Max lack?

• Power Autonomy, Robustness, and Reliability

  • MOOG Actuators

  • HPU (on-board)

  • Power Supply (on-board)

• 3rd generation of the HyQ flagship robots

• Quadrupedal Robots for Heavy Duty Operations

C. Semini et al, Brief introduction to the quadruped robot HyQReal, I-RIM, 2019.

HyQReal (2019-)

HyQReal (2019-)

• First steps of HyQReal

• Before pulling the plane

Tele-manipulation with HyQReal (2021)

• Robot Teleoperativo (Partners: IIT + INAIL*)

• Separate humans from hazardous situations

• Teleoperating the robot

  • Hand exoskeleton and a VR interface (pilot station)

  • Teleoperate a robot with an arm (field robot)

*National Institute for Insurance Against Accidents at Work

Tele-manipulation with HyQReal (2021)

INAIL-IIT Robotic Arm

• ARM:

  • Custom designed to fit the project's requirements

  • 5 DOF robotic arm

  • Leight weight: 5kg excluding end-effector

  • Payload: 5kg excluding end-effector

  • Torque/Impedance Control

  • Optimized Workspace

    • Pick up

    • Operate

    • Stow (for better distributed weight when not used)​

• HERI-II hand

  • 4-fingered

  • Integrated pressure sensing

  • Integrated visual feedback

E. Barrett et al., A Compliant Mechanism with Progressive Stiffness for Robotic Actuation, AIM, 2021.
Z. Ren et al., HERI II: A robust and flexible robotic hand based on modular finger design and under actuation principles, IROS, 2018.

REMOTArm + HEXOTrAc-Plus

I. Sarakoglou et al., HEXOTRAC: A highly Under-Actuated Hand Exoskeleton for Finger Tracking and Force Feedback, IROS, 2016.
A. Brygo et al., Synergy-Based Interface for Bilateral Tele-manipulations of a Master-Slave System with Large Asymmetries, ICRA 2016.
G. Li et al., A Novel Orientability Index and the Kinematic Design of the RemoT-ARM: A Haptic Master with Large and Dexterous Workspace, 2020.

REMOTArm + HEXOTrAc-Plus

Robot Teleoperation

Overview of the DLS lab's Robots

Hip Adduction-Abduction (HAA)

Hip Flexion-Extension (HFE)

Knee Flexion-Extension (KFE)

Legged Robots are moving out from research labs

into the real world.

Is there anything common in these applications?

The terrains that legged robots traverse are never the same.

Legged Robots Have to be Terrain Aware.

Terrain Aware Locomotion (TAL) means that the robot is able to take decisions based on the surrounding terrain using its perception.

Terrain Aware Locomotion (TAL)

• Proprioceptive TAL (PTAL)

• Exteroceptive TAL (ETAL)

Proprioceptive TAL (PTAL)

M. Focchi et al., Local Reflex Generation for Obstacle Negotiation in Quadrupedal Locomotion, CLAWAR, 2013.
V. Barasuol et al., On the detection and localization of shin collisions and reactive actions in quadruped robots, CLAWAR, 2019.

Exteroceptive TAL (ETAL)

V. Barasuol et al., Reactive Trotting with Foot Placement Corrections through Visual Pattern Classification, IROS, 2015.
O. Villarreal et al., Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs, RA-L, 2019.

Proprioceptive TAL vs. Exteroceptive TAL

PTAL is crucial when

• Visual feedback is denied (smoky areas, or areas with thick vegetation). 

• Terrain map is unreliable.

But

• Limited to corrective actions (step reflexes).

• Do not act on what is ahead of the robot.

Let legged robots take decisions based on the surrounding terrain using their perception.

Outline

• Part I: Proprioceptive Terrain-Aware Locomotion

  • Passive Whole-Body Control (pWBC)

  • STANCE: Locomotion Adaption over Soft Terrain

• Part II: Exteroceptive Terrain-Aware Locomotion

  • ViTAL: Vision-base Terrain-Aware Locomotion

Whole-Body Control (WBC)

• To achieve dynamic locomotion, the robot should:

  • Track the desired reference trajectories.

  • Maintain balance.

  • Reason about the robot's (full) dynamics and limits.

• To achieve TAL, the robot should reason about the contact interaction such as the terrain's inclination and frictional properties, etc.

• How to deal with all these objectives in an optimal way?

• A Whole-Body Control (WBC) Framework.

Whole-Body Control (WBC) Framework

• Locomotion Planner [1]-[3]

• State Estimator [4]

• WBC

• Low-level Control [5]

[1] B. Aceituno-Cabezas et al., Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimization, RA-L 2017.
[2] C. Mastalli et al., Trajectory and Foothold Optimization using Low-Dimensional Models for Rough Terrain Locomotion, ICRA, 2017.
[3] M. Focchi et al., Heuristic planning for rough terrain locomotion in presence of external disturbances and variable perception quality, STAR, 2020.
[4] S. Nobili et al., Heterogeneous sensor fusion for accurate state estimation of dynamic legged robots, RSS, 2017.
[5] T. Boaventura et al., Modelbased hydraulic impedance control for dynamic robots," T-RO, 2015.

• Execute the planned trajectories of the body and swinging leg (control tasks).

• Keep the robot balanced.

• Respect the un-actuated part of the dynamics. 

• Respect the robot's actuated part of the dynamics, and the joint & torque limits.

• Respect the contact constraints due to the interaction with the terrain.

• Maintain contact consistency.

Whole-Body Control (WBC) Objectives

• Cast these objectives as an optimization problem via Quadratic Programming (QP).

• Map the solution to joint torques using inverse dynamics.

Results: pWBC

B. Aceituno-Cabezas et al., Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimization, RA-L 2017.
C. Mastalli et al., Trajectory and Foothold Optimization using Low-Dimensional Models for Rough Terrain Locomotion, ICRA, 2017.

Recap: pWBC

• The pWBC is aware of the terrain's geometry and friction.

• The pWBC can achieve dynamic locomotion while compliantly balancing the robot's trunk.

• We showed the capabilities of the WBC under different gaits and different rigid terrain conditions.

• Now, is the WBC terrain aware?

S. Fahmi et al., Passive Whole-Body Control for Quadruped Robots: Experimental Validation over Challenging Terrain, RA-L, 2019.

• Most of WBCs are not terrain-aware. They fail to generalize beyond rigid terrain.

• Locomotion over soft terrain by itself is challenging because of the induced contact dynamics.

What is the problem?

• Mismatch between what the WBC assumes vs. what is actually happening.

• Drop the assumption of the rigid terrain. Use a more generic model that accounts for the terrain impedance.

• But that’s not enough, how do you know the impedance parameters of the terrain. 

• Can we, estimate it?

How to solve it?

Passive WBC (pWBC) = Standard WBC (sWBC)

Reformulate the sWBC to account for the contact dynamics

C3 = compliant contact consistent

How will the new WBC know about the current terrain impedance parameters?

C3 = compliant contact consistent

?

Feedback the terrain impedance parameters using a Terrain Compliance Estimator (TCE)

C3 = compliant contact consistent

TCE

Simultaneously, at every control loop

WBC Framework

Results: STANCE

Results: STANCE

Results: STANCE

Results: STANCE

• Unlike the state of the art WBC, we do not assume that the ground is rigid.

• Adapt online to any type of terrain compliance.

• With STANCE, HyQ can traverse and transition between multiple terrains with different compliance without pre-tuning.

• Each leg independently senses and adapts to changes in terrain compliance.

• STANCE makes the locomotion strategy C3. It allowed HyQ to remain consistent with ground, and thus more robust in challenging and scenarios.

Recap: STANCE

S. Fahmi et al., STANCE: Locomotion Adaptation over Soft Terrain, T-RO, 2021.

Outline

• Part I: Proprioceptive Terrain-Aware Locomotion

  • Passive Whole-Body Control (pWBC)

  • STANCE: Locomotion Adaption over Soft Terrain

• Part II: Exteroceptive Terrain-Aware Locomotion

  • ViTAL: Vision-base Terrain-Aware Locomotion

Locomotion Planning

Two ways to deal with locomotion planning:

• Plan the feet & base states simultaneously.

• Separate the plan into feet & base plans.

• Planning the feet motion ⇒ foothold selection

• Planning the body motion ⇒ pose adaptation

Esteban 2020

Villarreal 2019

Barasuol 2015

Foothold Selection

V. Barasuol et al., Reactive Trotting with Foot Placement Corrections through Visual Pattern Classification, IROS, 2015.
O. Villarreal et al., Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs, RA-L, 2019.
D. Esteban et al., On the Influence of Body Velocity in Foothold Adaptations for Dynamic Legged Locomotion via CNNs, CLAWAR, 2020.

What is the problem?

Pose Adaptation

• The goal is to find the optimal robot's pose that can:

  • Maximize the reachablity of the selected footholds.

  • Avoid terrain collision and maintain static stability.

• An pose optimizer to deal with these objectives.

M. Kalakrishnan et al., Learning, planning, and control for quadruped locomotion over challenging terrain, IJRR 2011.
P. Fankhauser et al., Robust Rough-Terrain Locomotion with a Quadrupedal Robot, IROS, 2018.

Kalakrishnan 2011

Fankhauser 2018

The Problem with Pose Adaptation

• The problem with current pose adaptation strategies is that they focus on finding one optimal solution based on given selected footholds.

• First, the footholds are selected.

• Then, the pose is optimized based on that.

• What if the given footholds are not reached (if the robot gets disturbed)?

• The robot may end up in a state with no reachable safe footholds. 

How to Solve This Problem?

Vision-Based Terrain Aware Locomotion (ViTAL)

• ViTAL is an online vision-based locomotion planning strategy.

• ViTAL selects footholds based on the robot's skills

• Also, find the robot pose that maximizes the chances of success in reaching these footholds.

• The notion of safety & success emerges from skills that characterize the robot's capabilities.

Vision-Based Foothold Adaptation (VFA)

Vision-Based Pose Adaptation (VPA)

Results: Performance with Different Velocities

Preliminary Experiments

Conclusion: ViTAL

• ViTAL allows HyQ and HyQReal to traverse a wide variety of terrains under various forward velocities.

  • These terrains included stairs, gaps, etc.

  • Forward velocities varied from 0.2m/s to 0.75 m/s.

• We showed that the VPA resulted in body poses that are aware of the terrain, and of what the robot and its legs can do.

• The VPA puts the robot in a pose that provides the feet with higher number of safe footholds which allows the robot to succeed.

Claudio Semini

Victor Barasuol

Geoff Fink

Michele Focchi

Andreea Radulescu

Octavio Villarreal

Domingo Esteban

Carlos Mastalli

Pointers: things I did not talk about

• Model Predictive Control

  • N. Rathod et al., Mobility-enhanced MPC for Legged Locomotion on Rough Terrain, under review, 2021.

  • O. Villarreal et al., MPC-based Controller with Terrain Insight for Dynamic Legged Locomotion, ICRA, 2020.

• Feasible Region

  • A. Abdalla et al., An Efficient Paradigm for Feasibility Guarantees in Legged Locomotion, under review, 2021.

  • R. Orsolino et al., Feasible Region: an Actuation-Aware Extension of the Support Region T-RO, 2020.

• State Estimation

  • S. Fahmi et al., On State Estimation for Legged Locomotion over Soft Terrain IEEE Sensors Letters, Jan. 2021.

  • G. Fink and C. Semini, Proprioceptive Sensor Fusion for Quadruped Robot State Estimation, IROS, 2020.

• Robot Balancing

  • C. Gonzalez et al., Line Walking and Balancing for Legged Robots with Point Feet, IROS, 2020.