How to evaluate surgical dataset quality and avoid leakage

Which quality checks matter most?

Use several layers of review. File checks can establish that media decodes and matches a manifest. Content review assesses whether the footage shows the intended procedures. Clinical review examines whether labels and interpretations are appropriate. Provenance review establishes what is known about the collection and its permitted use.

The point of this separation is practical: an issue needs the right owner. Re-encoding a video will not resolve a licensing gap, and relabeling a phase will not create missing visual coverage.

How does leakage happen in surgical video?

A single operation can produce many highly similar frames and multiple exported clips. If these are randomly scattered across training and test sets, the test may measure familiarity with a case rather than performance on a new case. Duplicate exports and near-duplicate edits can create the same problem.

Plan grouping before splitting. Use suitable internal identifiers to keep related material together, then document the separation relevant to the question being tested. A site-held-out evaluation and a case-held-out evaluation answer different questions; neither label should be used without describing the actual construction.

The IMDRF: good machine learning practice, 2025 calls for appropriate independence of training and test data, considering dependencies involving patients, sites, and acquisition. That principle supports a deliberate split design rather than a convenient frame-level shuffle.

Why can more hours still leave important gaps?

A collection may contain many repeated examples of a narrow procedure, recording system, or visual condition. That can be useful for one question while failing another. Ask for distributions relevant to the intended use and identify which characteristics are known, unavailable, or inferred.

Review exclusions. If low-visibility segments, atypical anatomy, or interrupted cases were removed, document the selection process. A polished dataset can accidentally hide the situations that will challenge a model in practice. Do not equate cleanliness with representativeness.

Treat missing metadata as a limitation to manage, not an invitation to infer sensitive attributes from video. Some coverage questions cannot be answered without appropriately collected contextual information.

What should a dataset acceptance scorecard contain?

• A specific requirement and its rationale.
• The review method: automated, sampled human review, or complete review.
• The unit and sample size used for the check.
• A pre-agreed acceptance criterion where appropriate.
• Observed findings, known exceptions, and an accountable owner.
• A decision: accept, remediate, exclude, or change the scope.

Keep metrics interpretable. A percentage without a denominator or sampling method can be hard to act on. If a small review sample finds a recurring defect, expand the relevant check rather than treating the initial average as definitive.

Record changes between releases. A corrected label or removed file can affect an experiment even when the total dataset size barely changes. A reproducible study needs a stable reference to the exact release it used.

Does a high-quality dataset prove a model is safe?

No. Data quality is one input to a broader development and evaluation program. A dataset can be well documented and still be unsuitable for a particular intended use. Likewise, a model can perform well on an offline benchmark without establishing performance in a clinical workflow.

Define the claim you want to make, the evidence needed for that claim, and the limits of the current experiment. Use the procurement checklist to connect those decisions to the supplier’s deliverables before purchase.