Coding agents use execution feedback—such as console logs and camera images—to assess task completion and revise their programs. This feedback may leave task outcomes unseen or failure causes unclear.
Conventional task-program refinement loopFly-by-Code (ours) adds Active Feedback to the loop
Click Active Feedback, View, or Probe to learn more.
Task program
ExecutionView · visual evidence
When the evidence gathered during task execution leaves part of the outcome unseen, the agent writes a diagnostic program that flies the robot to new viewpoints and returns the images with its notes. What it sees helps the agent decide whether the task is done and how to revise the program.
Another View case (2-Objects Pick-and-place): Did it land in the basket? ↓
Probe · physical evidence
When execution feedback leaves a failure cause unclear, the agent writes a diagnostic program to test possible explanations through targeted physical interactions in simulation. Starting from the state left by the task program, it can test how an object responds to contact, how a door moves, or whether a load can be lifted. These tests provide evidence to help the agent diagnose the failure and revise the task program.
Another Probe case (Align): Why does the box keep turning? ↓
Active Feedback
Active Feedback is a diagnostic stage between task execution and the decision to finish or revise the task program in a simulation-based refinement loop. The agent uses the same robot APIs to write a separate diagnostic program targeting questions left unresolved by execution feedback alone. The returned evidence supports completion assessment and failure diagnosis, guiding subsequent program revisions.
Separating diagnosis from task progress lets the agent choose observations and physical tests for their information value rather than their immediate contribution to task completion.
Task “Take the water bottle from the cabinet across the hall and put it in the basket hanging on the stairs.”
Task programs refined in simulation run on our physical aerial manipulator through shared robot APIs, without changes to the task-program source code. They use real-world observations to guide execution rather than replaying simulated trajectories. In our experiments, these programs repeatedly completed two multi-stage tasks despite differences between the simulated and physical scene layouts and variations in the robot’s initial position and heading.
Open both cabinet doors, grasp the bottle inside, and carry it to the hanging basket.
Pick the hammer from the pegboard and place it in the basket, then carry the wrench to the upper shelf.
A custom omnidirectional coaxial quad-tiltrotor carries a four-degree-of-freedom arm, a UMI gripper and a wrist RGB-D camera. The videos show successful runs. In the table below, denominators include all evaluated trials per task, even those not reaching the subtask.
| Task | Outcome | Successful trials |
|---|---|---|
| 2-Objects Pick-and-place | Grasp hammer | 5/5 |
| Place hammer in basket | 4/5 | |
| Grasp wrench | 5/5 | |
| Place wrench on shelf | 5/5 | |
| Full task | 4/5 | |
| Cabinet | Open left door | 5/5 |
| Open right door | 4/5 | |
| Grasp bottle | 4/5 | |
| Place bottle in basket | 3/5 | |
| Full task | 3/5 |
Six aerial manipulation tasks in two simulated environments, with the same coding-agent configuration and task-execution limit. Each trial permits one initial execution and up to ten program revisions; 10 trials per task and condition.
Across 60 trials, AF makes 5 false completion declarations, compared with 25 for CaP-X and 19 for CaP-X+Trace.
| Method | Per task (TP / FP / ND) | Overall | ||||||
|---|---|---|---|---|---|---|---|---|
| Cabinet | 1-Obj P&P | 2-Obj P&P | Payload | Align | Sort | TP / FP / ND | Precision | |
| CaP-X | 1/1/8 | 9/1/0 | 4/5/1 | 3/6/1 | 0/10/0 | 2/2/6 | 19/25/16 | 43.2% |
| CaP-X+Trace | 0/3/7 | 10/0/0 | 6/3/1 | 6/3/1 | 3/7/0 | 4/3/3 | 29/19/12 | 60.4% |
| Ours (CaP-X+AF) | 4/1/5 | 10/0/0 | 9/1/0 | 10/0/0 | 7/3/0 | 5/0/5 | 45/5/10 | 90.0% |
TP: correct completion declaration. FP: incorrect declaration. ND: no declaration before the step limit. Precision = TP / (TP + FP).
Two recorded simulation trials, with excerpts from the agent’s analysis, code and console.
View / Look againDid it land in the basket?New viewpoints reveal a failed placement.Recorded simulation case. Code, console output, and agent quotations are excerpts; omissions are marked […]. Image notes are inspection questions written before capture.
hammer done: True [...] wrench done: True
“The previous program reports both parts done, but neither is confirmed.”
Hammer. […] “That shows the hammer left the hand, not that it landed inside the basket. It could have dropped beside the basket or onto the stairs. I need a view looking down into the basket.”
Wrench. […] “I need to look at the upper shelf more closely from above, then at the lower shelf and the floor in front of the cabinet, and write down where the wrench actually is.”
def look(target, offsets, note): target = np.asarray(target, float) for off in offsets: vp = target + np.asarray(off, float) if go(vp, target - vp): return get_observation(note=note) # [...] fl = np.array([3.55, cc[1], 0.05]) o = look(fl, [[-0.8, 0, 1.0], [-1.0, 0.3, 1.1]], "Floor in front of the cabinet's open face: did the wrench fall to the floor?") report(o, ["wrench"])
[wrench] score 0.91 [...] med [ 3.567 0.953 -0.009] [...] [hammer] score 0.92 [...] med [ 2.463 3.979 -0.009] [...]
REGENERATE — the next program adds a release-and-hold routine, revises the hammer release pose, and tries deeper wrench placements.
def release_and_hold(): open_gripper() p = pos() goto_pose(p, settle=True) # hold still ~3 s so the object drops free get_observation() open_gripper()
The trial succeeded on the fourth task execution. The case shows AF producing evidence against a reported completion and directing the revision.
Probe / Test a hypothesisWhy does the box keep turning?Contact tests guide the next pushing strategy.Recorded simulation case. These are sequential diagnostic observations, not a controlled contact-location ablation. Code and console excerpts retain their original values.
After repeated pushes kept rotating a carton, the agent used AF to test alternative contact locations and sideways slides.
“My working guess is sideways friction drag.” […]
“I need controlled experiments to learn how much each action twists the carton: centre push, high-edge push, low-edge push, and a push followed by a small sideways slide while still in contact.”
The program tested these actions sequentially, returning to the same viewpoint after each probe to measure the response from RGB-D observations. In the code and console below, “hi-edge” and “lo-edge” are the carton’s left and right edges in the head-on camera view, and “drag +y” / “drag -y” are small slides to the left / right.
open_gripper() # [...] exps = [("centre", 0.0, 0.0), ("hi-edge", +1, 0.0), ("lo-edge", -1, 0.0), ("centre+drag+y", 0.0, +0.015), ("centre+drag-y", 0.0, -0.015), ("hi-edge", +1, 0.0)] for name, where, drag in exps: # [...] r = go(G, approach=N, z_approach=0.15, allow_contact=True, settle=True) # [...] if drag != 0.0 and r: G2 = G + np.array([0.0, drag, 0.0]) go(G2, approach=N, allow_contact=True, settle=True) # [...] # [...] f2, s2 = measure_B("after " + name)
RESULT centre: centre moved 0.0285, yaw change -3.17 deg RESULT hi-edge: centre moved 0.0385, yaw change -0.31 deg RESULT lo-edge: centre moved 0.0241, yaw change -4.01 deg RESULT centre+drag+y: centre moved 0.0439, yaw change -4.56 deg RESULT centre+drag-y: centre moved 0.0426, yaw change -4.75 deg RESULT hi-edge: centre moved 0.0430, yaw change -2.11 deg
The first left-edge push produced less rotation per unit of advance than the centre and right-edge pushes. Neither sideways slide reversed the rotation.
REGENERATE — the next program initialized a linear push-response model from the diagnostic observations and used it to select the lateral contact position.
# ---------------- twist model: yaw change (deg) per cm of push = alpha + beta*u ---------------- # u = contact offset from face centre / half-width (+ toward +y). Prior from observed pushes: # [...] w = y2 - y1; yc = 0.5 * (y1 + y2) # [...] alpha, beta = twist_model() # [...] u = float(np.clip((rate_des - alpha) / beta, -0.82, 0.82)) yp = yc + u * w / 2.0 # [...] if dc_cm > 0.4: twist_data.append((u, dpsi / dc_cm, min(2.0, dc_cm / 2.0)))