Behavior Cloning

Part 1

MIT 6.421

Fall 2026, Lecture 8

Russ Tedrake

Evaluation

Is checkpoint A better than checkpoint B?

  • "Violin plots" from Bayesian analysis assuming a uniform prior
    • "compact letter display" for separation

Rigorous evaluation

  • Real-world hardware testing
    • A/B testing. Always blind, randomized trials.
    • Rich reporting (not just "pass/fail").
  • Extensive simulation-based testing
    • Many more experiments, and they are repeatable
    • Many are skeptical of simulation; our investment has paid off

Multitask Scaling Laws

from the LBM 1.0 paper

more data from this task

more data from other tasks

fine-tuning

pretraining

Failure analyses for our hardest (real) skills

ClearKitchenCounter

Single task

LBM finetuned

The post(s) that caused the buzz

Note: It did take 2.5 minutes to run..., so it's a harbinger, but not quite ready to ship...

“Pick up the red block from the table and place it inside the bowl” using Inspect Robots

{
  "model": "provider/model",
  "messages": [
    {
      "role": "system",
      "content": "You are controlling a robot through tool calls. Each observation gives you the current state and camera images. Work toward the goal in small motions; re-check the observation after every motion. Every move call needs a note. Respond with exactly one tool call per turn..."
    },
    {
      "role": "user",
      "content": "Goal: pick up the red cube"
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Current observation.\nstate[eef_pose]: x=0.31 y=0.12 z=0.40 ...\n"
        },
        {
          "type": "text",
          "text": "camera 'front':"
        },
        {
          "type": "image_url",
          "image_url": { "url": "data:image/png;base64,..." }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "move_to",
        "description": "Move to absolute Cartesian end-effector targets...",
        "parameters": {
          "type": "object",
          "properties": {
            "targets": {
              "type": "object",
              "description": "Map of dimension name to value. Valid names: ..."
            },
            "note": {
              "type": "string",
              "description": "What you observe and why you chose this motion."
            }
          },
          "required": ["targets", "note"]
        }
      }
    }
  ]
}
{
  "tool_calls": [
    {
      "id": "call_123",
      "type": "function",
      "function": {
        "name": "move_to",
        "arguments": "{\"targets\":{\"x\":0.30,\"y\":0.10,\"z\":0.25},\"note\":\"The gripper is above and to the side of the cube, so I’m moving over it before lowering.\"}"
      }
    }
  ]
}

Example VLM prompt:

Example VLM response (tool call):

{
  "targets": {
    "x": 0.30,
    "y": 0.10,
    "z": 0.25
  },
  "note": "The gripper is left of the cube, so I’m moving above it before lowering."
}