Introduction: Anatomy of a

Manipulation System

MIT 6.4210/2: Robotic Manipulation

Fall 2026, Lecture 1

Today

  • What do we mean by "manipulation"?
  • Logistics 
  • Anatomy of a manipulation system
  • Drake

Matt Mason says

  • Definition 1 (etymological). Manipulation refers to the activities performed by hands.
  • ...
  • Definition 5. Manipulation refers to an agent’s control of its environment through selective contact.

Matthew T. Mason.  Toward Robotic Manipulation.  Annual Review of Control, Robotics, and Autonomous Systems, 1:1-28, 2018.

Autonomous "Open-world" Manipulation

  • Definition 5. Manipulation refers to an agent’s control of its environment through selective contact.

 

  • Even broader: "Open-world" manipulation requires:
    • rich perceptual understanding of the environment,
    • "common-sense" understanding of objects,
    • the ability to make long-term (task-level) plans, and combine them with fine (joint-level) motions.

Course Info

Website: http://manipulation.mit.edu/Fall2026

Course Info

Website: http://manipulation.mit.edu/Fall2026

6.4210 is a CI-M (Communication Intensive in the Major)

Communications Instructors from Comparative Media Studies/Writing

Recitations Fridays (starting Sept 18)

Use of AI

  • For communication tasks

  • Read the full policy

Use of AI

  • For homework and project implementation- unlimited

  • But we expect you to understand what is generated.   Exam questions based on material.

  • Read the policy

6.4210 vs 6.4212

  • Higher expectations on 6.4212 projects
  • 6.4212 has extra problems on psets and quizzes

Logistics I

  • Make sure you're on Piazza and Gradescope
  • CI-M (6.4210) material on Canvas.
  • CI-M (6.4210) recitation signup on Canvas (ignore Registrar)
  • Recorded lectures via Canvas
     
  • Read the lecture notes
  • ~Weekly problem sets
  • Midterm quizzes in class 10/14, 11/16 - [9-11am]
  • Final project (groups of 3):
    • Group Report
    • Group Video and
    • Individual Summary (in-class questions - 12/9)

Logistics II - Problem sets

  • Problem sets out on Thursday, due following Thursday​
  • You can find link to Pset repo on Schedules page
  • Preferably install software on your machine.  Using Colab is also supported.
  • Solutions are submitted via Gradescope (separate Gradescope for grad/undergrad)

TODO

Autonomous "Open-world" Manipulation

  • Definition 5. Manipulation refers to an agent’s control of its environment through selective contact.

 

  • Even broader: "Open-world" manipulation requires:
    • rich perceptual understanding of the environment,
    • "common-sense" understanding of objects,
    • the ability to make long-term (task-level) plans, and combine them with fine (joint-level) motions.

Dexterity vs Generality

VAMP-MR: Huang et. al

Basic components

  • Links

  • Joints (Revolute, Prismatic)

  • Sensors

    • Proprioception (position, torque)

    • Force and/or tactile

    • Vision: RGB(D), Laser Scan

  • Actuation

    • Motors (electric, hydraulic)

    • Transmission (gear reduction, backlash, friction)

    • Motor drivers (position control, torque control)

Exploring robot arms

https://www.infineon.com/cms/en/discoveries/robotics-systems-safety-security/

https://spectrum.ieee.org/robotics/industrial-robots/rethink-robotics-baxter-robot-factory-worker

Universal Robots

Flexiv

Kuka iiwa

Kinova Jaco

ABB Yumi

Franka Panda

Schunk WSG 50

Force-controlled planar gripper

Sandia Hand

iHY (i-Robot, Harvard, Yale)

Robotiq 3-fingered gripper

Allegro Hand

Shadow hand

Hands

 

Mobile Manipulators

Mobile manipulators

Rainbow RBY1

Humanoids (w/ dexterous hands)

Figure

This class

  • Robot arms.
    • The hardware matters.
    • We'll often use Kuka iiwa
  • Robot hands.
    • We'll often use Schunk WSG​​​​​​​

The anatomy of a  manipulation system

To be clear: we're not just controlling the arm

state of the robot x state of the environment

Low-level feedback control from cameras / sensors

(continuous time/state/action)

Scene-understanding +

Task-level Planning

(discrete/symbolic)

Robot System Architecture

From Kris Hauser - Robotic Systems (online)

Levine*, Finn*, Darrel, Abbeel, JMLR 2016 

Visuomotor policies

perception network

(often pre-trained)

policy network

other robot sensors

learned state representation

actions

Manipulation is much more than "pick and place"!

But even pick and place is challenging in the general case

C. Garrett, C. Paxton, T. Lozano-Perez, L. Kaelbling, D. Fox 2020

What is Drake?

Drake features

  • Systems framework
  • Deterministic simulations
  • Optimization framework
  • Sophisticated contact modeling (hydroelastic)

Signals and Systems Abstraction

Denny Freeman

State

Systems Framework

Drake models computations as Systems, wired together in a Diagram.

  • LeafSystem: user-defined input/output and dynamics.
  • Diagram: composition of multiple subsystems.
  • Context: stores state, time, parameters, inputs.
  • Ports: inputs and outputs that connect systems.

Example: Simple LeafSystem

class DoubleSystem(LeafSystem):
	def __init__(self):
		super().__init__()
		self.DeclareVectorInputPort('u', 1)
		self.DeclareVectorOutputPort('y', 1, self.CalcY)
        
	def CalcY(self, context: Context, output: BasicVector) -> None:
		u = self.get_input_port('u').Eval(context)
		output.SetFromVector([2*u[0]])
  • a = BasicVector(3)    # 3-d vector

  • a.SetFromVector(array)    # from numpy array

  • a.CopyToVector()    # to numpy array

Diagram Composition

  • Diagrams allow composition of multiple subsystems.

    - Each system has input/output ports.

    - Ports are connected to build larger systems.

builder = DiagramBuilder()
builder.AddSystem(sys1)
builder.AddSystem(sys2)
builder.Connect(sys1.get_output_port(0), sys2.get_input_port(0))
diagram = builder.Build()

Dynamic systems

class VerticalMassUnderGravity(LeafSystem):
    """
    Continuous-time 1D vertical mass:
        ż = v
        v̇ = g + f/m   (constant gravity)
    State vector: x = [z, v]
    """
    def __init__(self, gravity=-9.81, mass=1.0):
        super().__init__()
        self._g = float(gravity)
        self._m = float(mass)

        # Continuous state: q (z), v (ż)
        self.DeclareContinuousState(2)
        # Input force f
        self.DeclareVectorInputPort("force", 1)

        # Output: position
        self.DeclareVectorOutputPort("position", 1, self.CalcPosition)

    def CalcPosition(self, context: Context, output: BasicVector) -> None:
        x = context.get_continuous_state_vector()
        output.SetFromVector([x[0]])

    def DoCalcTimeDerivatives(self, context:Context, derivatives:ContinuousState) -> None:
        x = context.get_continuous_state_vector().CopyToVector()
        f = self.get_input_port(0).Eval(context)
        z, v = x

        z_dot = v
        v_dot = self._g + float(f)/self._m

        derivatives.get_mutable_vector().SetFromVector([z_dot, v_dot])

Simulation

if __name__ == "__main__":
    
    builder = DiagramBuilder()
    mass_sys = builder.AddSystem(VerticalMassUnderGravity())

    # Two ways of doing the logger
    logger = LogVectorOutput(mass_sys.get_output_port(0), builder)
    # logger = builder.AddSystem(VectorLogSink(1))
    # builder.Connect(mass_sys.get_output_port(0), logger.get_input_port(0))

    diagram = builder.Build()

    ctx = diagram.CreateDefaultContext()
    sim = Simulator(diagram, ctx)

    ctx.get_mutable_continuous_state().SetFromVector([10.0, 0.0])
    sys_ctx = mass_sys.GetMyContextFromRoot(ctx)
    mass_sys.get_input_port(0).FixValue(sys_ctx, 10.0)

    sim.AdvanceTo(2.0)  # simulate 2 seconds
    z_final = mass_sys.get_output_port(0).Eval(sys_ctx)
    
    print("Final position [z] =", z_final)

    plot(sim)

Drake's Physics Engine: MultibodyPlant

MultibodyPlant is Drake's class for modeling robots and mechanisms.

  • Add bodies, joints, actuators, forces.
  • Register geometry with SceneGraph for visualization/collision.
  • Parse models from URDF/SDF files.
  • Requires Finalize() after model is built to produce System
builder = DiagramBuilder()
plant, scene_graph = AddMultibodyPlantSceneGraph(builder, time_step=1e-4)
parser = Parser(plant, scene_graph)
parser.AddModelsFromUrl("package://drake_models/iiwa_description/urdf/iiwa14_primitive_collision.urdf")
plant.Finalize()
diagram = builder.Build()

Contact simulation