Part 3:
MIT 6.421
Robotic Manipulation
Fall 2023, Lecture 10
Follow live at https://slides.com/d/82eG46E/live
(or later at https://slides.com/russtedrake/fall23-lec10)
class PlannerState(Enum):
WAIT_FOR_OBJECTS_TO_SETTLE = 1
PICKING_FROM_X_BIN = 2
PICKING_FROM_Y_BIN = 3
GO_HOME = 4
class Planner(LeafSystem):
def __init__(self, plant):
LeafSystem.__init__(self)
self.DeclarePeriodicUnrestrictedUpdateEvent(0.1, 0.0, self.Update)
self._mode_index = self.DeclareAbstractState(
AbstractValue.Make(PlannerState.WAIT_FOR_OBJECTS_TO_SETTLE))
...
def Update(self, context, state):
mode = context.get_abstract_state(int(self._mode_index)).get_value()
...
state.get_mutable_abstract_state(
int(self._mode_index)).set_value(PlannerState.PICKING_FROM_X_BIN)
class Planner(LeafSystem):
def __init__(self, plant):
...
self._traj_X_G_index = self.DeclareAbstractState(
AbstractValue.Make(PiecewisePose()))
self._traj_wsg_index = self.DeclareAbstractState(
AbstractValue.Make(PiecewisePolynomial()))
...
def Update(self, context, state):
...
if mode == PlannerState.WAIT_FOR_OBJECTS_TO_SETTLE:
if context.get_time() - times["initial"] > 1.0:
self.Plan(context, state)
...
def Plan(self, context, state):
...
X_G, times = MakeGripperFrames(X_G, t0=context.get_time())
traj_X_G = MakeGripperPoseTrajectory(X_G, times)
traj_wsg_command = MakeGripperCommandTrajectory(times)
state.get_mutable_abstract_state(int(
self._traj_X_G_index)).set_value(traj_X_G)
state.get_mutable_abstract_state(int(
self._traj_wsg_index)).set_value(traj_wsg_command)
I would group them into:
http://www.ai.mit.edu/projects/leglab/robots/robots.html
1. approach, s.t. mug is in view. 2. visual servo 3. insert to grasp 4. retract 5. move to pre-place 6. place (force termination)
work by Siyuan Feng et al. at TRI
1. general approach the plate, s.t. it's roughly in view for the wrist camera 2. visual servo for alignment 3. assuming known geometry and well alignment wrt the plates, close the fingers to the right amount, insert one finger tip between the plates. 4. push the top plate towards one side of the sink with the finger tip. (terminate with force) 5. kind of scoop the gripper into a solid grasp, while pushing onto the plate (terminate with force) 6. retract and go to pre-place configuration. 7. insert to place. (terminate with force)
work by Siyuan Feng et al. at TRI
Relative to state machines:
https://github.com/RobotLocomotion/6-881-examples/blob/km_thesis_work/bt_planning/behavior_tree.py
https://en.wikipedia.org/wiki/Stanford_Research_Institute_Problem_Solver
/// Base type for all action primitives
class ActionPrimitiveInterface
{
virtual bool IsCandidate(const State& state) const = 0;
virtual Container GetOutcomes(const State& state) const = 0;
virtual double EstimateActionCost(
const State& current, const State& target) const = 0;
virtual Container Execute(const State& state) = 0;
virtual double Ranking() const = 0;
};
OpenDishwasherDoor CloseDishwasherDoor StartDishwasher LoadSoapPacket PullOutLowerRack PullOutUpperRack PullOutSilverwareRack PushInLowerRack PushInUpperRack PushInSilverwareRack
PutPlateInDishwasher ReorientMug PutMugInDishwasher OptimisticPutMugInDishwasher PutSilverwareInDishwasher PushUnmanipulableDish SenseDishCounts SenseDishToManipulate
work by Calder Phillips-Grafflin et al. at TRI
DishTaskState(const DishState& active_dish_state,
int32_t clean_items_put_away, int32_t clean_items_in_dishwasher,
int32_t dirty_items_in_dishwasher,
int32_t dirty_items_available,
const DishwasherState& current_dishwasher_state);
DishState(const RigidTransformd& dish_pose, DishType dish_type,
DishProgress dish_progress, RackRequirement rack_required);
DishwasherState(bool known, bool door_open, bool lower_rack_out,
bool upper_rack_out, bool silverware_rack_out,
bool lower_rack_full, bool upper_rack_full,
bool silverware_rack_full, bool started, bool soap_loaded);
enum class DishType : uint8_t {
Unknown = 0x00,
Mug = 0x01,
Plate = 0x02,
Silverware = 0x03
};
enum class RackRequirement : uint8_t {
Unknown = 0x00,
LowerRack = 0x01,
UpperRack = 0x02,
SilverwareRack = 0x03
};
enum class DishProgress : uint8_t {
Unknown = 0x00,
Unmanipulable = 0x01,
LikelyManipulable = 0x02,
Sensed = 0x03,
OptimisticOriented = 0x04,
Oriented = 0x05,
Placed = 0x06,
Released = 0x07
};
work by Calder Phillips-Grafflin et al. at TRI
We replan before each action to handle unexpected outcomes...
from https://www.cs.toronto.edu/~sheila/2542/s14/A1/introtopddl2.pdf
Example: Gripper task with four balls
There is a robot that can move between two rooms and pick up or drop balls with either of his two arms. Initially, all balls and the robot are in the first room. We want the balls to be in the second room.
(define (domain gripper-strips)
(:predicates (room ?r)
(ball ?b)
(gripper ?g)
(at-robby ?r)
(at ?b ?r)
(free ?g)
(carry ?o ?g))
(:action move
:parameters (?from ?to)
:precondition (and (room ?from) (room ?to) (at-robby ?from))
:effect (and (at-robby ?to)
(not (at-robby ?from))))
(:action pick
:parameters (?obj ?room ?gripper)
:precondition (and (ball ?obj) (room ?room) (gripper ?gripper)
(at ?obj ?room) (at-robby ?room) (free ?gripper))
:effect (and (carry ?obj ?gripper)
(not (at ?obj ?room))
(not (free ?gripper))))
(:action drop
:parameters (?obj ?room ?gripper)
:precondition (and (ball ?obj) (room ?room) (gripper ?gripper)
(carry ?obj ?gripper) (at-robby ?room))
:effect (and (at ?obj ?room)
(free ?gripper)
(not (carry ?obj ?gripper)))))
https://github.com/SoarGroup/Domains-Planning-Domain-Definition-Language/blob/master/pddl/gripper.pddl
(define (problem strips-gripper4)
(:domain gripper-strips)
(:objects rooma roomb ball1 ball2 ball3 ball4 left right)
(:init (room rooma)
(room roomb)
(ball ball1)
(ball ball2)
(ball ball3)
(ball ball4)
(gripper left)
(gripper right)
(at-robby rooma)
(free left)
(free right)
(at ball1 rooma)
(at ball2 rooma)
(at ball3 rooma)
(at ball4 rooma))
(:goal (and (at ball1 roomb)
(at ball2 roomb)
(at ball3 roomb))))
https://github.com/SoarGroup/Domains-Planning-Domain-Definition-Language/blob/master/pddl/gripper-4.pddl
and then came the
Experiments with GPT-4
for task-level planning
Scenario:
Scooping vegetables
courtesy of Charlie Richter et al from TRI
Perception system (or a human!) summarized environment state
The planner only reasoned about the discrete state / actions
but it's only a matter of time...
Certainly! Here's a step-by-step guide to clean the whiteboard using the items in the image:
1. **Preparation**:
- Clear the whiteboard of any magnets or attached papers.
- Make sure the whiteboard is free of any obstructions.
2. **Dry Erase Marker**:
- Sometimes, drawing over old marks with a dry erase marker and then erasing can help remove old stains. Use the red dry erase marker in the picture to do this if necessary.
3. **Lysol or Surface Cleaner**:
- Spray a small amount of the Lysol or the other surface cleaner onto a soft cloth or tissue. Do not spray directly onto the whiteboard as it might be too harsh for the board's surface.
4. **Wiping**:
- Gently wipe the whiteboard in circular motions, ensuring that you're not applying too much pressure. This will help in removing the marker stains.
5. **Drying**:
- After wiping with the cleaner, use a separate dry tissue or cloth to dry the whiteboard. This will prevent any streaks or residue.
6. **Tissue Boxes**:
- The tissue from the boxes can be used both for applying the cleaning solution and for drying. Make sure you use a fresh tissue when switching from cleaning to drying.
7. **Final Check**:
- Once you've wiped down the entire board and dried it, check for any leftover marks or streaks. If any remain, repeat the cleaning process until the board is clean.
Remember:
- Always be gentle when cleaning to prevent any scratches or damage to the whiteboard.
- Avoid using abrasive materials or too much cleaning solution.
- After cleaning, always dry the board thoroughly to maintain its smooth surface and prevent any ghosting from old markers.
Once you've finished cleaning, your whiteboard should be ready for use again!