{"id":"a333c471-aec8-499e-91c2-2ee517c3240a","arxiv_id":"2412.10594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark and fine-tuned models show that multi-task training improves average perceptual-similarity accuracy on known tasks but does not generalize to out-of-distribution perceptual tasks.","lead":"UniSim-Bench combines 25 existing datasets into 7 perceptual-similarity tasks to measure how well AI models match human notions of similarity. The authors find that general-purpose models beat task-specific metrics on unseen tasks, and that a multi-task fine-tuned model (UniSim) wins on average on the core tasks but still fails on out-of-distribution tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-hoc selection of the IQA/PAA scoring protocol on the test set (App. C) can inflate the averaged accuracies supporting the headline ranking; the central claim needs a fixed or validation-selected protocol to stand.","rationale":"I read the paper as making a benchmark-plus-model contribution: UniSim-Bench and UniSim models, with the main quantitative claims being (1) UniSim achieves highest average accuracy on core 2AFC tasks and generalizes to held-out datasets, and (2) specialized metrics underperform general-purpose CLIP outside their training domain. The most defensible strong point is the consistent per-task improvements and the public code and models. The weakest load-bearing point is not the benchmark's task commensurability—the authors are explicit that the benchmark is an open-ended collection, and most per-task conclusions are robust to averaging choices—but the post-hoc selection of the IQA/PAA evaluation protocol on the test set. This selection directly affects the 'Avg' column that supports the headline. It is a concrete, fixable methodological issue rather than a fatal flaw. The reader's weakest_assumption differs (commensurability), though their rationale mentions the selection issue; I count this as partial agreement. A fixed-protocol re-run would settle whether the concern changes the ranking. If the result is robust, the paper's conditional acceptance stands; if not, the paper needs substantial qualification. Hence I recommend the verdict remain CONDITIONAL (UNCHANGED relative to the reader).","tokens_in":25181,"tokens_out":5583,"duration_ms":52955,"concrete_test":"Recompute Table 1 (and Table 2 PAA) for all encoder-based models twice: once using only the naive 'A high quality photo.' reference-similarity protocol for IQA/PAA, and once using only the CLIP-IQA good/bad softmax protocol. Then recompute per-task averages and the final 'Avg' column. If UniSim remains top-ranked and the specialized-vs-CLIP comparisons hold under both fixed protocols, the concern is resolved; if the ranking changes or the margin collapses, the paper must report both protocol variants or select the protocol on a held-out validation split before testing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In App. C ('Evaluation setup'), the authors state that for encoder-based models on IQA and PAA they test two scoring protocols—the naive reference-prompt similarity and the CLIP-IQA good/bad softmax—and 'report the results of the one which performs best on average on the task.' This selection is made on the same test data used to compute Tables 1 and 2, with no validation split and no multiple-comparison correction. The headline claim is about average accuracy, and IQA is one of the four core tasks contributing to that average; UniSim's largest margins over CLIP appear in IQA and Text-2AFC. Because the reported per-task averages are maxima over two protocols rather than a fixed metric, the ranking of models in the 'Avg' column is not an evaluation under a single, pre-specified procedure. The bias is not necessarily directional, but it is uncontrolled: models differ in how much they benefit from the chosen protocol, and the choice is made after seeing test performance. This is load-bearing because the central relative claim (UniSim highest average; specialized models underperform CLIP) rests on these averaged numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UniSim-Bench, a benchmark of seven multi-modal perceptual tasks built from 25 existing datasets, and uses it to evaluate general-purpose vision-language models, task-specialized perceptual metrics, and the authors' proposed multi-task models UniSim (CLIP-based and LLaVA-NeXT-based). The main empirical claims are that specialized perceptual metrics often underperform general-purpose CLIP models on tasks outside their training domain, that a single multi-task fine-tuned model (UniSim) achieves the highest average accuracy on the core 2AFC tasks, and that all current models generalize poorly to the more distant OOD tasks. The paper also includes ablations on IT-2AFC training data and on scaling the number of alternatives in NAFC evaluation.","tokens_in":25477,"tokens_out":11128,"duration_ms":103422,"significance":"If the evaluation issues are resolved, this is a useful contribution: it provides a broad, openly available benchmark for perceptual similarity metrics and a clean multi-task training recipe that improves over single-task specialists while identifying their generalization limits. The paper is unusually thorough in its baseline coverage, and the authors release code and models, which strengthens reproducibility. The central claims are empirical rather than derived, so the main risk is not circularity but evaluation bias; the protocol-selection issue described below is the key concern that must be addressed before the headline rankings can be trusted.","major_comments":[{"comment":"The evaluation protocol for IQA and PAA is not fixed: for every encoder-based model, the authors test two scoring protocols (reference-prompt cosine similarity and the CLIP-IQA good/bad softmax) and report whichever 'performs best on average on the task.' This selection is made on the same test data used to compute Tables 1 and 7 and the PAA part of Tables 2 and 8, with no validation split and no correction for multiple comparisons. Because IQA contributes to the headline 'Avg' column and PAA contributes to the OOD average, the reported per-task numbers are maxima over two different metrics rather than the output of a single evaluation procedure. This is load-bearing for the central ranking claims: different models may benefit from the chosen protocol to different degrees, so the comparison of UniSim with CLIP and with specialized IQA models (e.g., LIQE, C2S) is not made under a single fixed rule. Please either (i) fix one protocol for all encoder models, (ii) select the protocol on a validation split, or (iii) report results separately under both protocols and demonstrate that the main conclusions are unchanged.","section":"Appendix C, 'Evaluation setup'; Tables 1 and 2"},{"comment":"The headline metric is a simple average of per-task average accuracies, but the tasks are heterogeneous in label semantics, difficulty, and chance level (2AFC, 3AFC, and retrieval). The paper states that the tasks are 'specific instances of a broader challenge' (Sec. 3.2), but it does not justify why equal weighting across tasks is the correct aggregation, nor does it show robustness to alternative aggregations (e.g., per-dataset averaging, chance-normalized accuracy, or leaving out a single task). Since the main conclusion that UniSim is the best unified metric and that specialized models underperform CLIP is stated in terms of this average, please add a robustness analysis of the aggregation choice; this would also help absorb the effect of the protocol-selection issue raised above.","section":"Section 3.2 and Tables 1-2 (average column)"}],"minor_comments":[{"comment":"The retrieval metric is not defined; please specify whether the reported numbers are mAP, recall@k, or another measure, and state how the per-dataset retrieval results are averaged into the task-level 'Avg'.","section":"Table 8, IR rows"},{"comment":"For datasets such as AGIQA-3K and KONIQ-10K, test samples are created by random pairing; please provide the exact pairing seed or release the generated pairs so that the benchmark results are exactly reproducible.","section":"Appendix B.1 and Table 4"},{"comment":"The sentence 'UniSim ranks as the first or second best across nearly all tasks' is not accurate for IQA: UniSim ViT-L/14 (87.0) is behind LIQE (88.2) and C2S (90.2); please rephrase or qualify this claim.","section":"Table 1"},{"comment":"Superscripts are used both to identify training datasets for specialized models and to label datasets in the table header, which makes the legend confusing; please use distinct notation for these two roles.","section":"Tables 1 and 7"},{"comment":"Please state explicitly that y is the index of the more similar alternative (y=0 for z0, y=1 for z1), because the sign of the hinge loss depends on this convention and the current text is ambiguous.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid empirical contribution and the protocol-selection issue is fixable without changing the scope of the work. If the authors can show that the main rankings are stable under a fixed IQA/PAA protocol (or under validation-based protocol selection), I would be willing to support acceptance. The benchmark and released models are valuable to the community; the key is to ensure the comparative evaluation is based on a single, pre-specified procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a genuinely useful benchmark paper, with one methodological wrinkle that needs attention. The core contribution is UniSim-Bench: 25 datasets repackaged into seven 2AFC-style tasks, with standardized evaluation and OOD splits, plus UniSim, a multi-task fine-tuned CLIP/LLaVA family. That integration is new—prior benchmarks cover narrower task sets—and the paper does a lot right: extensive evaluation across 20+ models, detailed per-dataset tables, released code and models, and a clean ablation in Table 3. The qualitative finding that specialized metrics like DreamSim and HPSv2 fall behind vanilla CLIP on tasks outside their training domain is likely robust and worth knowing.\n\nThe soft spot is exactly where the reader put it. Appendix C says that for IQA and PAA, the authors test two scoring protocols per encoder model and report whichever performs best on average on the task. That is test-set selection. It means the per-model numbers in Tables 1 and 2 are not a single fixed evaluation, and the averaged ranking in the 'Avg' column is partly a max over protocols. It is not necessarily fatal: the UniSim margins on IQA are large, and the headline ranking would probably survive a fixed protocol. But for a paper whose product is a benchmark, this is a real integrity issue. Benchmarks need frozen evaluation rules. The fix is easy: pick one protocol, or select it on a validation split, and report both variants.\n\nThe second issue, also noted by the reader, is the averaging of seven heterogeneous tasks into one 'Avg' column. That is defensible as a summary, but it deserves a sensitivity analysis or per-task discussion rather than a single mean. Smaller quibbles: no error bars, even on tiny test sets like IR (70 queries) and ImageReward (412 examples), and some LMM comparisons are incomplete because the models cannot do retrieval.\n\nOverall, this deserves a serious referee. The benchmark is a potential community resource, the models are reproducible, and the core findings are probably right. But the protocol-selection problem must be resolved before the numbers can be trusted as a ranking. I would send it to peer review with a request to fix the evaluation protocol and add error bars.","headline":"Useful benchmark paper with a genuine protocol-selection flaw in Appendix C that needs fixing before the ranking claims can be trusted.","tokens_in":25941,"tokens_out":2214,"would_cite":true,"duration_ms":21795,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single multi-task perceptual metric, UniSim, beats task-specific specialists on average across a new 7-task, 25-dataset benchmark, while all existing metrics still fail to generalize to unseen task types.","keywords":["perceptual similarity metrics","multi-modal benchmark","two-alternative forced choice","vision-language models","CLIP fine-tuning","multi-task learning","generalization","human perception of similarity"],"falsifier":"Recompute the benchmark's task averages after normalizing each dataset by its chance-level difficulty or after calibrating each model separately per task; if plain CLIP then matches or beats UniSim on the average, the claim that multi-task training itself improves perceptual alignment would be disproved. Alternatively, add a new held-out 2AFC dataset whose human labels come from pairwise preference rankings rather than corruption or quality scores: if UniSim's advantage over its CLIP base vanishes there, its edge is task-specific rather than a general perceptual ability.","tokens_in":24970,"feed_emoji":"🎯","tokens_out":6885,"duration_ms":55617,"temperature":0.7,"pith_summary":"This paper argues that human perception of similarity should not be carved into separate narrow tasks, and that a single metric can be trained to handle several of them at once. To show this, it assembles UniSim-Bench, a benchmark of seven multi-modal similarity tasks drawn from 25 datasets, and measures how well general-purpose models (CLIP, LLaVA-NeXT, Mantis), task-specialized metrics (DreamSim, HPSv2, ImageReward, PAC-S, LIQE), and its own multi-task fine-tuned models (UniSim) align with human judgments. The central finding is that specialized metrics often lose to plain CLIP on tasks outside their training niche, while UniSim, fine-tuned simultaneously on four two-alternative forced-choice (2AFC) tasks, attains the highest average accuracy on the core tasks and generalizes to held-out datasets within those tasks. The paper also shows that none of these models generalizes well to truly unseen task types, which it takes as evidence that learning a unified perceptual metric that captures human similarity remains an open problem.","feed_headline":"Unified metric outranks specialists on 7 perceptual tasks","feed_subtitle":"Benchmarking 25 datasets shows specialists lag behind general models; multi-task fine-tuning lifts the average.","key_machinery":"The engine of the paper is the reduction of heterogeneous perceptual tasks to a common two-alternative forced choice (2AFC) format: every data point is a triplet (reference, alternative A, alternative B) plus a label saying which alternative humans find more similar to the reference, higher quality, or better aligned. This format lets UniSim-Bench compare all models on the same classification metric and lets the authors fine-tune a CLIP image encoder with a single hinge loss over all four core tasks simultaneously, keeping the text encoder frozen and using LoRA to limit overfitting. For the generative variant, the same 2AFC samples are converted into natural-language instructions and used to fine-tune LLaVA-NeXT, whose multi-image and image-text interleaved handling suits the triplet structure. The benchmark's train/held-out split is the third piece of machinery: by withholding specific datasets and whole task types from training, it defines three tiers of generalization—standard, intra-task, and inter-task—that structure all the conclusions.","core_discovery":"The paper claims that a single multi-task fine-tuned perceptual metric, UniSim, achieves the highest average accuracy on the core 2AFC tasks of UniSim-Bench while also generalizing to held-out datasets within those tasks, and that specialized task-specific metrics frequently underperform general-purpose CLIP models on tasks outside their training domain. Concretely, UniSim models fine-tuned from CLIP (ViT-B/32 and ViT-L/14) and from LLaVA-NeXT-0.5B all rank first or second on nearly every core task and beat their own base models on every task, while DreamSim, ImageReward, HPSv2, PAC-S, and LIQE each show large drops when evaluated outside the task they were trained for. On out-of-distribution tasks (perceptual attribute assessment, odd-one-out, retrieval), however, both specialized and unified metrics fall below general-purpose baselines, except for near-OOD attribute assessment where multi-task training gives a small gain.","pith_inferences":["Because the benchmark averages tasks with equal weight, conclusions about which model wins could shift under a difficulty-weighted or application-weighted aggregate; this is an inference the paper does not explore.","The standardized 2AFC triplet structure suggests a natural extension: the same training recipe could be applied to other backbones (SigLIP, BLIP-2, larger LMMs) and to new datasets as they appear, effectively turning UniSim-Bench into a continuously growing evaluation instead of a fixed one.","If perceptual metrics are used as training rewards for generative models, the finding that specialists underperform general backbones out-of-domain implies that reward models should be evaluated on the target task's actual data distribution rather than only on the benchmark they were tuned on.","The near-OOD gain on perceptual attribute assessment hints that a metric trained on a broader set of tasks may transfer to simple photometric or geometric attributes; adding more attribute types beyond brightness, colorfulness, contrast, and sharpness would test this."],"forward_implications":["Specialized perceptual metrics should not be assumed to generalize: a metric trained for one task, such as text-to-image alignment, can fall below its own generic backbone on a closely related task such as caption-to-image alignment.","Multi-task fine-tuning on 2AFC data yields a single metric that beats both its base model and most specialists on the average of the core tasks, so unified training is a viable route rather than an obvious sacrifice in accuracy.","Intra-task generalization is achievable: UniSim improves on held-out datasets within a task it trained on, sometimes by large margins, as on CD-COCO and AGIQA-3K.","Inter-task generalization remains unsolved: on out-of-distribution tasks (PAA, OOO, IR), neither specialized nor unified perceptual metrics beat general-purpose models on average, indicating that current training fits task structure rather than a universal notion of similarity.","When the number of alternatives in IT-2AFC grows from 2 to 8, all models lose accuracy, but the CLIP-based UniSim retains roughly three times the accuracy of its base model, suggesting the unified training transfers some robustness to harder comparisons."],"supporting_citations":[{"why":"Provides the CLIP encoder backbones and the general-purpose baselines; its cosine-similarity function is the similarity measure that UniSim-CLIP fine-tunes.","marker":"[39]"},{"why":"Supplies the LLaVA-NeXT multi-image instruction-tuned model that becomes UniSim-LL-N, along with the M4-Instruct data mixed into its training.","marker":"[27]"},{"why":"Contributes the NIGHTS dataset for the Img-2AFC task and the DreamSim specialized model, the strongest image-similarity baseline UniSim must match.","marker":"[8]"},{"why":"Supplies the HPDv2 dataset for IT-2AFC and the HPSv2 specialized model, a key example of a specialist that underperforms its CLIP backbone outside its task.","marker":"[48]"},{"why":"Contributes the ImageReward dataset for IT-2AFC and the ImageReward metric, another specialist baseline that shows poor out-of-domain transfer.","marker":"[49]"},{"why":"Provides the PAC-S Text-2AFC specialized metric used as a baseline and as evidence of limited cross-task generalization.","marker":"[40]"},{"why":"Supplies the LIQE IQA-specialized metric, which excels on its own task but degrades on unseen perceptual tasks.","marker":"[55]"},{"why":"Contributes the BAPPS dataset used for Img-2AFC evaluation and the LPIPS perceptual metric context that UniSim-Bench builds on.","marker":"[54]"}],"fun_headline_variants":["Unified metric tops specialists on average but not on novel tasks","Multi-task fine-tuning wins average perceptual similarity","UniSim benchmark reveals unified metric edge, OOD weakness","General model beats specialized metrics on 7 perceptual tasks","Unified perceptual metric: best average, but still limited generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes the seven tasks, assembled from datasets with different annotation protocols, label meanings, and difficulty, all measure one underlying 'human perception of similarity,' so a simple average accuracy across tasks is a meaningful ranking of how well a model matches human perception.","fun_headline_variants_meta":{"raw":{"variants":["Unified metric tops specialists on average but not on novel tasks","Multi-task fine-tuning wins average perceptual similarity","UniSim benchmark reveals unified metric edge, OOD weakness","General model beats specialized metrics on 7 perceptual tasks","Unified perceptual metric: best average, but still limited generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1611,"prompt_tokens":989,"completion_tokens":622,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":605,"tokens_out":622,"duration_ms":6264,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:48:00.629813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the benchmark's task averages after normalizing each dataset by its chance-level difficulty or after calibrating each model separately per task; if plain CLIP then matches or beats UniSim on the average, the claim that multi-task training itself improves perceptual alignment would be disproved. Alternatively, add a new held-out 2AFC dataset whose human labels come from pairwise preference rankings rather than corruption or quality scores: if UniSim's advantage over its CLIP base vanishes there, its edge is task-specific rather than a general perceptual ability.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP encoder backbones and the general-purpose baselines; its cosine-similarity function is the similarity measure that UniSim-CLIP fine-tunes."},{"cited_title":"Dream- sim: Learning new dimensions of human visual similarity using synthetic data","cited_arxiv_id":null,"evidence_quote":"Contributes the NIGHTS dataset for the Img-2AFC task and the DreamSim specialized model, the strongest image-similarity baseline UniSim must match."},{"cited_title":"Imagere- ward: Learning and evaluating human preferences for text- to-image generation","cited_arxiv_id":null,"evidence_quote":"Contributes the ImageReward dataset for IT-2AFC and the ImageReward metric, another specialist baseline that shows poor out-of-domain transfer."},{"cited_title":"Positive-augmented contrastive learning for image and video captioning evaluation","cited_arxiv_id":null,"evidence_quote":"Provides the PAC-S Text-2AFC specialized metric used as a baseline and as evidence of limited cross-task generalization."},{"cited_title":"Blind image quality assessment via vision- language correspondence: A multitask learning perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the LIQE IQA-specialized metric, which excels on its own task but degrades on unseen perceptual tasks."},{"cited_title":"Efros, Eli Shecht- man, and Oliver Wang","cited_arxiv_id":null,"evidence_quote":"Contributes the BAPPS dataset used for Img-2AFC evaluation and the LPIPS perceptual metric context that UniSim-Bench builds on."}],"review_version":1}