{"id":"50b8d8c0-4d65-4f6b-a9b1-790ff0007b82","arxiv_id":"2506.04803","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SupeRANSAC, a well-engineered combination of known RANSAC components, achieves state-of-the-art accuracy on homography, fundamental and essential matrix, and rigid and absolute pose estimation benchmarks.","lead":"This paper presents SupeRANSAC, a unified RANSAC pipeline that combines existing robust estimation techniques into a single system. It reports consistent accuracy improvements over many established frameworks across five geometric vision problems on 11 public datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tuning on test pairs is a real contamination risk; before taking 'consistently highest accuracy' at face value, the paper needs a disjoint-split rerun and per-dataset breakdowns.","rationale":"Good-faith reading: the paper is an engineering/empirical contribution whose central claim is that a single RANSAC pipeline beats problem-specific baselines across five geometry tasks. For that claim to hold, the evaluation must be free of selection bias. I agree with the reader that the tuning-on-test protocol in Sec. IV.c is the weakest link: it is explicit, unquantified, and directly affects the metric (AUC) used to support the headline. The concern is not an accusation; the author's 'unlikely' argument is reasonable for 1-2 parameters, and the public code plus ablations are genuine supporting evidence. But because reported margins are sometimes small (homography 0.41 vs 0.39) and no per-dataset or repeated-run statistics are provided, the leaked tuning pairs could matter even if they are only a few percent of the data. A disjoint-split rerun would settle it. A secondary observation is that Table V already shows SupeRANSAC is not the best on Aachen Day strict AUC (80.7 vs 88.5 and 88.1), so the phrase 'consistently highest accuracy' should be softened regardless. Verdict remains CONDITIONAL: the concern is real but testable, and the paper has enough independent evidence (code, ablations, large-scale evaluation) that it should not be rejected.","tokens_in":27674,"tokens_out":8315,"duration_ms":107760,"concrete_test":"Re-run the Table I/III evaluations with a strictly disjoint protocol: randomly split each dataset's pairs into a 200-pair tuning set and a non-overlapping test set; tune every method's parameters on the tuning set only; evaluate on the test remainder. Also report SupeRANSAC's AUC@10 separately on the 1200 tuning pairs vs. the held-out test pairs. If the held-out margins over the next best method remain at the reported level (≥5 AUC points for F/E, and ≥0.02 for H) and tuning-pair AUC is not systematically higher than held-out AUC, the leakage concern is resolved. If margins drop below the reported gains or tuning-pair AUC materially exceeds held-out AUC, the protocol is biasing the central comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.c states that 200 image pairs per dataset were randomly selected from the test sets and used to tune parameters (e.g., inlier threshold, spatial coherence weight), then 'these tuning pairs were not subsequently removed from the test set.' Because the central claim is a quantitative accuracy comparison (e.g., '6 AUC points on average' for fundamental matrix), any systematic upward bias in the tuned method's score on those pairs is directly load-bearing. The argument that 200 pairs are too few to overfit 1-2 parameters is plausible but unquantified: the reported tables aggregate over 39,592 pairs, so a small per-pair bias could shift the average by the same magnitude as the claimed margins on homography (0.41 vs 0.39 AUC@10, Table III). Moreover, no repeated-run variance or per-dataset breakdown is given, so the 'consistent highest accuracy' claim cannot be distinguished from a few favorable datasets. This does not by itself invalidate the contribution; it does mean the headline claim currently rests on an unmeasured assumption about leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SupeRANSAC, a unified RANSAC-style robust estimation pipeline that assembles existing components—PROSAC/P-NAPSAC sampling, sample and model degeneracy checks, MAGSAC++ scoring, GC-RANSAC-style and nested-RANSAC local optimization, IRLS final optimization, and task-specific minimal/non-minimal solvers—into a single framework for homography, fundamental and essential matrix, absolute pose, and rigid 3D registration. The central claim is that this engineered system consistently outperforms existing public frameworks (OpenCV, PoseLib, pyCOLMAP, GC-RANSAC, MAGSAC++, VSAC) across 11 large-scale datasets and two feature types, with headline improvements such as 6 AUC points on average for fundamental matrix estimation. The evaluation includes accuracy-runtime trade-offs, threshold sensitivity curves, and component ablations. Code is publicly released.","tokens_in":27879,"tokens_out":5271,"duration_ms":70798,"significance":"If the empirical claims withstand scrutiny, the paper makes a useful practical contribution: it demonstrates that a carefully engineered, problem-specific RANSAC pipeline can match or beat specialized estimators across multiple geometric tasks, and it provides a detailed qualitative account of which components matter. The experimental breadth is genuinely large: 39,592 image pairs for epipolar geometry, two feature types, a dedicated homography benchmark, visual localization benchmarks, and 3D registration, with ablations for scoring, local optimization, and final optimization. The public code release is a further strength. However, the evaluation protocol has a load-bearing weakness: hyperparameters were tuned on subsets of the test data without removing those subsets, and no error bars or significance tests are reported. Until the evaluation is rerun on a clean split, the headline 'consistently highest accuracy' claim is not fully supported.","major_comments":[{"comment":"The evaluation protocol has test-set leakage. Section IV.C states that 200 image pairs per dataset were randomly selected from the test sets, used to tune parameters, and then 'these tuning pairs were not subsequently removed from the test set.' This is a direct contamination risk for the central quantitative claim. The concern is not merely hypothetical: several reported margins are small, e.g., Table III (SP+LG) gives SupeRANSAC AUC@10 of 0.41 versus 0.39 for GC-RANSAC, and Table I gives 0.59 versus 0.53 for the next best method. A small systematic upward bias on the 200 tuned pairs per dataset could shift the aggregated averages by exactly this magnitude. The assertion that overfitting to 200 pairs is 'highly unlikely' is unquantified. Moreover, the ablation studies and threshold-sensitivity curves in Figs. 5-6 are computed on the same 1,200-pair tuning set, so they demonstrate internal consistency but not generalization. Please rerun the main comparison on a disjoint evaluation split (or remove tuning pairs from the reported aggregates), and provide per-dataset breakdowns so the reader can assess the size of any leakage effect.","section":"IV.C, Tables I-III, Figs. 5-6"},{"comment":"No error bars, confidence intervals, or statistical significance tests are reported for any of the main comparisons. RANSAC-based estimators are stochastic, and single-run AUC differences of 0.02-0.03 (e.g., Table III, SP+LG row: SupeRANSAC 0.41 vs. GC-RANSAC 0.39; Table IV: mAA 0.51 vs. 0.44) may be consistent with run-to-run variation. Because the paper's central claim is 'consistently highest accuracy,' the authors should report mean and standard deviation over multiple random seeds, or bootstrap confidence intervals, for at least the headline comparisons in Tables I-III, and should state how many repeats were used. Without this, the claimed margins cannot be distinguished from noise.","section":"Tables I-VI, Fig. 3"},{"comment":"The headline claim of 'consistently highest accuracy' is stronger than the data in several places. In Table V, on the Aachen Day subset SupeRANSAC achieves 80.7 at the (0.25m, 2°) threshold while LO-RANSAC [COLMAP] achieves 88.5, a clear regression on one of the two absolute-pose benchmarks. In Table VI, on 3DMatch SupeRANSAC's registration recall of 92.0 is below MAGSAC++'s 92.5, although its rotation/translation errors are better. The paper should either temper the claim to 'consistently competitive' or provide an aggregate summary measure across all tasks showing a net improvement, so that the one-RANSAC-for-all thesis is evaluated fairly.","section":"Abstract, Section IV.D, Tables V-VI"},{"comment":"The description of baseline tuning is under-specified. The text says parameters were 'optimized on this set' for all estimators, but it does not report the parameter ranges searched, the number of configurations evaluated per method, or the criterion used to select the final value. Since SupeRANSAC and the baselines have different scoring functions and coherence weights, the fairness of the comparison depends on these details. Please provide the tuning protocol (grid/range, number of evaluations, selection rule) for each method and problem type.","section":"IV.C"}],"minor_comments":[{"comment":"The abbreviations 'GC-RSC' and 'LO-RSC' are used for GC-RANSAC and LO-RANSAC without being defined in the tables; please harmonize with the names used in the text.","section":"Tables I, II, III, IV, VI"},{"comment":"Reference [25] lists a placeholder URL (gts.sourceforge.net) alongside the PoseLib GitHub link; the entry should be corrected.","section":"References"},{"comment":"The manuscript header states 'Manuscript received April 19, 2005,' which appears to be a typo for 2025.","section":"Header"},{"comment":"The equation for the optimistic score reuses s(k) for both the partial score and the optimistic score; a distinct symbol such as \\hat{s}(k) would avoid the confusing notation.","section":"III.G"},{"comment":"The axis labels in Figure 5 are poorly formatted, with items such as 'MAGSAC ACRANSAC' and '2.08 × 100' requiring careful reading; please clean up the label spacing and scientific notation.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The core issue is statistical rather than conceptual: the system is plausible and the ablation studies are informative, but the test-set contamination and lack of error bars undermine the strength of the headline claim. A rerun with a clean split and repeated seeds is a straightforward fix and should be a requirement for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a strong engineering paper that probably delivers most of what it claims, but the headline numbers aren't fully trustworthy until the author reruns with a disjoint train/test split. The tuning-on-test-set issue is real, and while I think the arithmetic suggests the bias is small, it still needs a clean fix.\n\nWhat's actually new: SupeRANSAC is a carefully assembled pipeline that combines PROSAC/P-NAPSAC sampling, MAGSAC++ scoring, GC-RANSAC local optimization, SP-RANSAC for homography, and problem-specific non-minimal solvers with Levenberg-Marquardt refinement. None of the components is new, but the systematic combination and per-problem configuration are. The paper's message—that the \"bells and whistles\" matter more than any individual algorithmic tweak—is useful and honestly argued. The ablations are thorough, the code is public, and the evaluation spans five geometry problems, two feature types, and 11 datasets. That is real evidence, more than most papers in this area bring.\n\nSoft spots. The tuning protocol is the main issue. Section IV.c explicitly says that 200 pairs per dataset were selected from the test sets, used to tune parameters, and not removed. The stress-test note worries this could explain the gains. I read the numbers differently: with 1200 tuning pairs out of 39,592, a per-pair bias would need to be ~0.5 AUC to explain even the 2-point homography margin, which is implausible. The more realistic risk is that a global threshold is overfit to a small, possibly unrepresentative set; since all baselines were tuned on the same set, the relative comparison is probably not badly biased. Still, the numbers are not an independent evaluation, and \"highly unlikely\" is not a substitute for a disjoint-split rerun. I would also ask for per-dataset breakdowns. The abstract's \"consistently highest accuracy\" is stronger than what Tables V and VI show: on Aachen Day, SupeRANSAC loses to COLMAP/PoseLib at the strict threshold, and on 3DMatch its recall is slightly below MAGSAC++. Minor inconsistencies, but they should be acknowledged.\n\nOne more note: several baselines come from the author's own implementations. They also report OpenCV versions where they exist, which is a reasonable safeguard. Self-citation is not the problem.\n\nThe absence of error bars is a minor concern; this subfield rarely reports them, and RANSAC variance is usually small at these iteration counts.\n\nWho it's for: anyone building or using a robust estimation library. It deserves a serious referee. My recommendation: accept with major revision, contingent on a clean-split rerun, per-dataset tables, and an abstract that matches the actual level of consistency.","headline":"A well-built unified RANSAC that likely works as advertised, but the test-set tuning leak and missing per-dataset numbers mean the headline gains need a clean-split rerun before I'd trust them.","tokens_in":28401,"tokens_out":5087,"would_cite":false,"duration_ms":58624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SupeRANSAC, a single unified RANSAC pipeline, is claimed to achieve the highest accuracy across homography, fundamental and essential matrix, rigid pose, and absolute pose estimation on 11 public large-scale datasets.","keywords":["RANSAC","robust estimation","homography estimation","fundamental matrix","essential matrix","absolute pose","rigid pose","MAGSAC++"],"falsifier":"Re-run all comparisons with the 200 tuning pairs per dataset excluded from the test set and with parameters tuned on a disjoint hold-out; if SupeRANSAC's reported AUC margins over the next-best method shrink or reverse, the central accuracy claim would be refuted.","tokens_in":1795,"feed_emoji":"🎯","tokens_out":1873,"duration_ms":85471,"temperature":0.7,"pith_summary":"The paper argues that the accuracy of RANSAC-style robust estimators is decided less by any single innovation such as sampling, scoring, or refinement, and more by how the full pipeline is assembled. It introduces SupeRANSAC, a single unified pipeline that combines PROSAC and P-NAPSAC sampling, MAGSAC++ scoring, GC-RANSAC local optimization, IRLS final optimization, and problem-specific minimal and nonminimal solvers. The central empirical claim is that this one configuration consistently reaches the highest accuracy on five geometric vision problems across 11 public large-scale datasets, improving on the best existing methods by, for example, 6 AUC points on average for fundamental matrix estimation. If correct, practitioners would not need to choose a different robust estimator per task, and the paper also implies that under-reported components such as local and final optimization matter more than the headline sampling or scoring method.","feed_headline":"One RANSAC pipeline tops five vision tasks","feed_subtitle":"SupeRANSAC reports top accuracy on five geometry tasks across 11 public datasets.","key_machinery":"The central object is the complete SupeRANSAC pipeline, whose load-bearing pieces are the scoring and optimization stages. The default scoring function is MAGSAC++, which marginalizes over a range of noise scales instead of committing to a single inlier-outlier threshold. Local optimization defaults to GC-RANSAC, which uses graph-cut to segment inliers from outliers followed by nested RANSAC sampling of size seven times the minimal sample, with a fallback to plain nested RANSAC when the correspondence count exceeds 2000. Final optimization is iteratively reweighted least squares with robust Cauchy weights, halving the threshold and re-evaluating consensus; for fundamental matrices, the nonminimal solver already integrates Levenberg-Marquardt refinement under the rank-2 parameterization, and the ablations show that adding a separate IRLS stage can slightly hurt. These stages carry the accuracy gains, as the ablation studies demonstrate for scoring, local optimization, and final optimization choices.","core_discovery":"The central claim is that SupeRANSAC, a unification of existing RANSAC improvements rather than a fundamentally new sampling or scoring principle, outperforms the state of the art on all five tested geometric estimation problems. The framework selects P-NAPSAC for spatially coherent problems such as homography, rigid pose, and absolute pose, and PROSAC for epipolar geometry to avoid degenerate localized samples. It applies explicit sample degeneracy checks such as twisted quadrilaterals and collinear 3D points, model checks such as determinant bounds and proper-rotation verification, MAGSAC++ as the default scoring function, preemptive verification with an optimistic score upper bound, GC-RANSAC local optimization that falls back to nested RANSAC above 2000 correspondences, and IRLS with Cauchy weights as final optimization. The paper reports the highest accuracy on fundamental and essential matrix estimation over six datasets totaling 39,592 image pairs, on homography over those datasets plus the HEB benchmark, on absolute pose on Aachen Day-Night and InLoc, and on rigid pose on 3DMatch and 3DLoMatch.","pith_inferences":["If the reported accuracy margins survive a fully disjoint tuning split, they suggest that SupeRANSAC's advantage comes from assembling known components carefully rather than from a new algorithmic idea, meaning other frameworks could adopt the same choices and close the gap.","The especially strong nighttime localization result suggests that threshold-robust scoring such as MAGSAC++ becomes more valuable when feature matching quality degrades, a testable hypothesis across other challenging conditions.","The deliberate choice of PROSAC over P-NAPSAC for epipolar geometry encodes a prior that spatial locality causes degeneracy in relative pose; a systematic study of when locality helps versus hurts could generalize sampling selection across problems.","Because the paper intentionally excludes learned outlier pruning, evaluating SupeRANSAC on top of learned pruning methods is a natural extension that may yield further gains or show that the pipeline's consensus reasoning makes them redundant."],"forward_implications":["One RANSAC configuration could replace problem-specific pipelines for homography, fundamental and essential matrix, rigid pose, and absolute pose estimation, yielding accuracy gains without task-specific code.","The combination of MAGSAC++ scoring, GC-RANSAC local optimization, and IRLS final optimization becomes a strong default recipe for future robust estimation libraries.","For fundamental matrix estimation, the built-in Levenberg-Marquardt refinement in the nonminimal solver is sufficient, so pipeline designers should not assume that adding more optimization stages always helps.","Preemptive verification using an optimistic score upper bound is broadly applicable to any scoring function with a per-point quality bound and does not degrade accuracy.","The reported gains imply that widely used existing implementations still leave significant accuracy on the table for common geometric estimation tasks."],"supporting_citations":[{"why":"Defines the RANSAC paradigm that SupeRANSAC extends and that all compared baselines share.","marker":"[8]"},{"why":"Provides the PROSAC quality-guided sampling used for epipolar geometry estimation.","marker":"[10]"},{"why":"MAGSAC++ scoring is the default scoring function of SupeRANSAC and a central accuracy driver.","marker":"[16]"},{"why":"Introduces locally optimized RANSAC, the lineage that SupeRANSAC's GC-RANSAC and nested-RANSAC local optimization builds on.","marker":"[18]"},{"why":"Supplies a holistic verified-consensus framework against which SupeRANSAC's accuracy is compared.","marker":"[24]"},{"why":"GC-RANSAC graph-cut local optimization is the default LO stage in SupeRANSAC for moderate-size inputs.","marker":"[27]"},{"why":"P-NAPSAC progressive spatial sampling is used for homography, rigid, and absolute pose estimation.","marker":"[33]"},{"why":"Hartley normalization of input correspondences is the preprocessing that stabilizes the solvers.","marker":"[53]"},{"why":"The HEB homography benchmark provides the training split and evaluation data for homography experiments.","marker":"[67]"},{"why":"GeoTransformer supplies the 3D correspondences used for rigid transformation estimation on 3DMatch and 3DLoMatch.","marker":"[74]"}],"fun_headline_variants":["SupeRANSAC unifies RANSAC tricks to win five tasks","One RANSAC to rule all five geometry tasks","SupeRANSAC: one pipeline, five state-of-the-art results","Unified RANSAC sweeps five vision benchmarks"],"cache_read_input_tokens":30592,"weakest_assumption_plain":"The claim rests on the assumption that tuning a handful of parameters on 200 image pairs per dataset and then evaluating on the full test set, without removing those pairs, does not inflate SupeRANSAC's measured accuracy.","fun_headline_variants_meta":{"raw":{"variants":["SupeRANSAC unifies RANSAC tricks to win five tasks","One RANSAC to rule all five geometry tasks","SupeRANSAC: one pipeline, five state-of-the-art results","Unified RANSAC sweeps five vision benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1666,"prompt_tokens":1007,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":586}},"tokens_in":623,"tokens_out":659,"duration_ms":8076,"temperature":1.0,"reasoning_tokens":586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:31:32.320221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run all comparisons with the 200 tuning pairs per dataset excluded from the test set and with parameters tuned on a disjoint hold-out; if SupeRANSAC's reported AUC margins over the next-best method shrink or reverse, the central accuracy claim would be refuted.","supporting_citations":[{"cited_title":"Matching with prosac-progressive sample consensus,","cited_arxiv_id":null,"evidence_quote":"Provides the PROSAC quality-guided sampling used for epipolar geometry estimation."},{"cited_title":"MAGSAC++, a fast, reliable and accurate robust estimator,","cited_arxiv_id":null,"evidence_quote":"MAGSAC++ scoring is the default scoring function of SupeRANSAC and a central accuracy driver."},{"cited_title":"Locally optimized RANSAC,","cited_arxiv_id":null,"evidence_quote":"Introduces locally optimized RANSAC, the lineage that SupeRANSAC's GC-RANSAC and nested-RANSAC local optimization builds on."},{"cited_title":"VSAC: Efficient and accurate estimator for h and f,","cited_arxiv_id":null,"evidence_quote":"Supplies a holistic verified-consensus framework against which SupeRANSAC's accuracy is compared."},{"cited_title":"Graph-cut RANSAC,","cited_arxiv_id":null,"evidence_quote":"GC-RANSAC graph-cut local optimization is the default LO stage in SupeRANSAC for moderate-size inputs."},{"cited_title":"In defense of the eight-point algorithm,","cited_arxiv_id":null,"evidence_quote":"Hartley normalization of input correspondences is the preprocessing that stabilizes the solvers."},{"cited_title":"A large- scale homography benchmark,","cited_arxiv_id":null,"evidence_quote":"The HEB homography benchmark provides the training split and evaluation data for homography experiments."},{"cited_title":"Geotransformer: Fast and robust point cloud registration with geometric transformer,","cited_arxiv_id":null,"evidence_quote":"GeoTransformer supplies the 3D correspondences used for rigid transformation estimation on 3DMatch and 3DLoMatch."}],"review_version":1}