Hi...

Confession

I know nothing about drones!!!

But !!!

As a roboticist, there is something that I have in common with you...

PAIN

...while debugging my application

"You should
log more!"

 

or...

My ongoing quest for
Observability and Debuggability

About Me

PlotJuggler

BehaviorTree

Humanoid
robotics

Localization
and mapping

Real-time
control

Manipulation

Perception

The most widely used debugging tool



     printf("here");
     
     // or 

     printf("value: %f\n", value);

What makes debugging difficult (for us)?

  • We can't use GDB-like debugging (breakpoints)
  • Multi-process systems are harder to debug
  • Working on embedded platforms with limited resources
  • We must not affect real-time performance

But most importantly...

We don't want to replicate errors, when they occurs

Its story time!

2013...

The primary technical goal of the DRC is to develop human-supervised ground robots capable of executing complex tasks in dangerous, degraded, human-engineered environments.

Competitors in the DRC are developing robots that can utilize standard tools and equipment commonly available in human environments, ranging from hand tools to vehicles.

The way the IHMC team developed their robotic software, blew my mind

  • Always log by default, not occasionally
  • Logged thousands of variables at 300 Hz
  • An in-house visualization tools to draw time series in multiple plots and windows

Logs were so detailed that most of the bugs could be found just inspecting them

What happened next is now history

PlotJuggler

Many plugins to ingest data from different sources.

Multi-transport and

multi-protocol

Processing functions:

low-pass filter, FFT, scale, integral, derivative or custom functions in LUA

Philosophy

Minimize the time from:

The moment you open your log

The moment when you have the answer to your question

Design criterias

How to minimize "time to insight":

  1. Intuitive drag & drop interface.
  2. A layout that makes it FAST to arrange your plots in multiple widgets.
  3. Focus on comparing data.
  4. Put the most used functions upfront, gently hide the least used ones.
  5. Don't clutter the interface!

The missing piece

We need
fearless, obsessive logging

The "ideal" data logging library

  • API should incentivize the user to add logged variables easily (low friction).
  • 1 million datapoints per second should be the norm.
  • Low-latency, on the side of the callee (asynchronous).
  • Must not affect real-time applications.
  • Space efficient: serialization with little to no overhead, preferably compressed.
  • Should potentially enable a rolling-buffer
  • Multi-sink: directly to file or published over ICP.

How close are we to that?

ROS/ROS2:

Consider pal_statistics

  • Good API
  • Little serialization overhead, but could be better...
  • Focused on ROS
  • Potentially the closest thing I know, to what I want

PX4:

Is ulog enough?

You tell me...

I am building my own logging library, to address these common requirements in a broader context and in a ROS-independent way

Conclusions

What was this talk even about?

  • A celebration of open source. PlotJuggler is my love letter to the robotic community
  • A reflection about the importance of observability and debuggability in our domains
  • A call to action: get involved if you think that this is important to you
Made with Slides.com