Xiaomi Releases PROVE: Perception-Aligned Object Removal Metrics
Xiaomi MiLM Plus introduces PROVE, an open-source Apache 2.0 evaluation harness featuring RC-S spatial and RC-T temporal video metrics.
Stock photo for illustration only, not from the actual event
- Xiaomi launches PROVE as a video object removal evaluation harness
- Features two core metrics: RC-S for spatial and RC-T for temporal analysis
- Requires PyTorch 2.6+, Python 3.10+, and Transformers 4.51+
- Includes benchmark datasets PROVE-M with 80 clips and PROVE-H with 100 hard clips
The Xiaomi MiLM Plus research team has released PROVE, an evaluation framework designed for video object removal, packaged as an Apache 2.0 PyTorch repository with a single command-line interface entry point via run_prove_metrics.py. The system relies on Python 3.10 or higher, PyTorch 2.6 or higher, Transformers 4.51 or higher, and DINOv2-giant weights. Additionally, binary masks are strictly required, where white pixels explicitly mark the removed object.
Object removal is fundamentally ill-posed and one-to-many, meaning numerous restorations remain plausible for the exact same hole, leaving no single unique ground truth. To address this, PROVE relies on local distribution matching within deep feature space instead of depending on global aggregation.
Evaluating video object removal is notoriously challenging in computer vision because minor temporal flickering or warping is immediately noticeable to human viewers. Frameworks like PROVE establish rigorous, mathematically grounded benchmarks that allow AI researchers to evaluate generative fill quality objectively rather than relying on subjective visual checks.
The framework utilizes two primary evaluation metrics built upon a shared conceptual foundation:
- RC-S (Spatial): Connected-component analysis splits the mask into independent targets. Each bounding box expands by one-third of its side length, crops are fed to DINOv2, and masks are downsampled to feature resolution. A w×w window slides across the feature map to compute squared Maximum Mean Discrepancy using a Gaussian RBF kernel between masked and local background features, averaging scores per target and then across targets.
- RC-T (Temporal): Adjacent frames are cropped jointly under the union of their masks to prevent misalignment, computing MMD solely inside the intersection representing regions restored across both frames. Ablation studies confirm that removing this cropping mechanism leaves RC-T completely blind to injected corruption.
Ablation tests isolate each individual contribution: DINOv2 achieves 0.59 average $ au$, outperforming DINOv3 at 0.51 and SAM at 0.44. Dropping the sliding window incurs a cost of 0.11, while substituting MMD with cosine similarity costs 0.07. Furthermore, RC-S stands as the most cost-effective spatial metric tested, operating 13.7 times faster than CFD.
Stock photo for illustration only, not from the actual event
The evaluation suite features PROVE-M, which contains 80 paired videos featuring tripod-captured inputs and target-free footage shot within a two-minute window, refined frame-by-frame using SAM3 masks, subjected to a three-stage quality control process, and augmented synchronously with Ken Burns-style motion. Each clip runs 81 frames at 1080p resolution. In addition, PROVE-H introduces 100 challenging unground-truth videos featuring crowds, flowing water, flames, textured terrain, multi-puddle reflections, and rapid motion, utilizing intentionally unrefined SAM3 masks.
On the public leaderboard, the 1.3B-parameter SVOR currently leads the combined RC-S category with a score of 0.5197, while EffectErase leads the RC-T metric at 0.2525.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment