Surgical video records what a camera can see. A robot-learning dataset may also need what the robot sensed, which action was issued, and how those signals align in time. Specify the model’s prediction target before deciding whether video alone is sufficient.
Video, state, and action are different records
| Record | What it describes | What it does not establish |
|---|---|---|
| Video observation | Images of the scene from a specified view. | The command sent to a robot, force at an instrument, or anatomy outside the view. |
| Robot state | Measured or estimated quantities such as joint position or tool pose, with units and a reference frame. | That the same values were the commands issued to the controller. |
| Robot action | The chosen prediction target, such as a command, pose increment, or other defined control representation. | The motion that was actually achieved, or whether the action was successful. |
| Task annotation | A human or algorithmic label for a phase, gesture, event, or outcome defined by a protocol. | A low-level robot command or a complete explanation of the scene. |
For supervised action prediction, the training target must be defined and available. Video-only methods may instead use representations, inferred actions, or another learning objective. State that choice explicitly. An inferred action should retain its derivation and validation record; it should not be described as a measured command.
Build a signal brief
Choose the research task and the proposed recording source. The planner lists requirements to resolve with the collection team.
Prediction target
Predict the specified robot action from observations available at decision time.
Signals to specify
- Task-visible video and any other observations the policy expects.
- Recorded or explicitly derived action targets, with units, ordering, coordinate frame, and control mode.
- Robot state where required, action/state timestamps, measured latency, and a shared time mapping.
- Episode boundaries, resets, interventions, failures, and a documented held-out evaluation.
Check the proposed source
A headset can capture the visible procedure and hand activity. It does not by itself record robot commands, measured robot state, or the internal endoscopic view.
Action supervision is unresolved. Scope an authorized robot-log integration or a method that explicitly learns from video without paired actions. Do not label video-only footage as paired control demonstrations.
Pilot acceptance
On a pilot, reconstruct an observation/action example and check timing, action semantics, missing signals, and whether future observations leak into the input.
The planner runs in your browser. It asks for no personal details and submits no selections. Its output is a starting specification, not a claim that these signals are available from Operavue.
A timing mistake that can hide in a clean table
Consider a synthetic example with a shared clock. The model predicts the command issued at 120 ms. A frame captured at 100 ms and available to the model at 112 ms could be an input. A frame captured at 133 ms could not have been available at that decision time, even if it is the nearest frame by capture time.
| Event | Capture or issue time | Available to the model | Use for a 120 ms decision? |
|---|---|---|---|
| Earlier camera frame | 100 ms | 112 ms | Potentially, if the other input requirements are met. |
| Target command | 120 ms | Prediction target | This is the action being predicted. |
| Later camera frame | 133 ms | 145 ms | No. It introduces future information in this example. |
Store timestamp semantics, clock mappings, and the alignment rule. Distinguish sensor capture, logging, receipt, and command issue times. Document dropped samples and resampling. Choose an acceptable alignment error for the task and test it; this example does not prescribe a tolerance.
For stereo, preserve timing and eye identity together. For state and action, record the coordinate frame, units, dimension order, and whether values are absolute, incremental, commanded, or measured. A column called “pose” is not enough.
What established projects make explicit
JIGSAWS pairs stereo endoscopic video with manipulator kinematics and gesture and skill annotations from bench-top tasks. Its documentation demonstrates why describing only the video misses much of the dataset. It does not establish that every kinematic channel is an action command. The release is for academic, noncommercial research.
The Open X-Embodiment project documents a specific action representation for its RT-X models, including the gripper frame and dimensions. That is a useful specification pattern, not a surgical control standard or evidence that its models transfer to surgery.
Hugging Face’s LeRobotDataset v3 documentation separates time-series data, camera video, and metadata. It records states, actions, and timestamps alongside episode metadata. A conversion to a named format still needs a validated mapping from the source signals; repackaging files cannot recover unrecorded commands.
What to include in the delivery specification
- Define the episode. Record its source procedure or session, boundaries, interruptions, and the relationship between clips.
- Define every signal. List its source, unit, shape, coordinate frame, sample timing, and missing-value behavior.
- Separate measurement from derivation. Keep raw references and document filtering, interpolation, estimated labels, and calibration changes.
- Protect the evaluation. Group related source cases before creating training, validation, and test sets. Choose additional site, operator, or device holdouts when the research question requires them.
- Test one complete example. Reconstruct the exact input and target a model would receive, then agree on measurable acceptance criteria before scaling capture.
Use the procurement brief to scope rights and delivery, the synthetic video manifest to map media, and the dataset comparison to check established research collections. Video metadata is only one part of a multimodal specification.
Where surgeon headset recordings fit
An optional headset may help capture visible hand activity and workflow from the surgeon’s perspective. Start by checking whether that perspective actually contains the target. Headset footage does not automatically contain an internal operative view, calibrated depth, or robot telemetry. Those requirements need separate scoping.
Operavue works with inquiries for existing recordings and new capture partnerships. A buyer should describe the research task and required signals; a surgeon or institution should describe the recording access they can authorize. Discuss a data requirement or explore recording as a surgeon.
FROM SPECIFICATION TO PARTNERSHIP
Scope the data your team needs.
Discuss existing footage, a custom capture program, or clinical annotation around a specific research task.
Share a non-confidential brief ↗