{"id":"efc2376b-ffb0-48c1-b7e6-d1c2deadbcd3","arxiv_id":"2501.15890","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Visual complexity is not captured by segmentation counts alone; multi-scale gradients, color diversity, and LLM-generated surprise scores improve prediction on several datasets, including a new Surprising Visual Genome set.","lead":"The paper tests why a simple two-feature computer model fails to predict human judgments of visual complexity, and shows that adding edge, color, and \"surprise\" scores improves accuracy on certain image datasets. It introduces a new dataset of surprising images and uses a large language model to score surprise, keeping the model interpretable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-dataset selection of MUC bit precision and of feature/dataset pairings uses the evaluation labels, so the reported gains are optimistic; a fixed-protocol rerun is needed to support the central claim.","rationale":"The reader's verdict is already CONDITIONAL, and the stress-test agrees that conditional acceptance is appropriate but sharpens the reason. The most load-bearing concern is not any single architectural choice but the repeated use of the evaluation labels to select the features, the bit precision, the datasets showcased, and the LLM that generates surprise scores. Because the central claim is explicitly about additional explained variance, even modest selection leakage directly inflates the very numbers that distinguish the proposed model from the Shen et al. baseline. The paper has real strengths: a new human-rated dataset, human validation of the LLM surprise scores, and extensive ablation studies. The qualitative claim that complexity is multifaceted is plausible and supported by participant reports. However, the quantitative increments in Table 1 should be read as upper bounds until the selection protocol is fixed. A concrete rerun with MUC bit precision held constant, or selected inside each training fold, would settle whether the MUC contribution survives; the same logic applies to pre-registering dataset-feature pairings and LLM choice. This does not change the reader's CONDITIONAL verdict, but it identifies the precise experiment that would allow acceptance.","tokens_in":18501,"tokens_out":4967,"duration_ms":53060,"concrete_test":"Recompute Table 1 using MUC bit precision fixed to 8 bits (and, as a robustness variant, 7 bits) for all datasets, keeping the same 3-fold cross-validation and repetition procedure, without per-dataset label-based selection. If the MUC improvements over sqrt(numseg)+sqrt(numclass) drop below or are no longer consistent with +0.08/+0.05, the central MUC claim is inflated by in-sample hyperparameter selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MSG, MUC, and Surprise add predictive variance beyond segment/class counts. The quantitative support is weaker than the tables suggest because several feature choices are made using the same complexity labels that are later used to measure performance. In the Multi-Scale Unique Color section the paper states: 'we report results using the best bit precision for each dataset in terms of correlation to complexity.' This is an in-sample choice over b in {3,...,8} on the full dataset before cross-validation, so the reported MUC gains (e.g., +0.08 on Sav. Art, +0.05 on Sav. Suprematism) are selected maxima, not unbiased estimates. The same pattern appears in dataset selection: the Methods say 'We select image subsets to demonstrate the effectiveness of our new features,' so VISC/IC9 Architecture for MSG and Sav. Art/Suprematism for MUC are chosen after observing which datasets benefit; and the surprise scorer GEMINI-1.5-FLASH was selected after comparing three LLMs on the SVG complexity labels (Appendix Table 3). Each individual choice may be small, but together they bias the headline increments upward and make the claim that the simple two-feature model is insufficient rest on optimistic effect sizes. The more general conclusion may survive, but the reported magnitudes are not trustworthy until evaluated under a fixed, label-independent feature-selection protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the interpretable two-feature complexity model of Shen et al. (2024) with three additional features: Multi-Scale Sobel Gradient (MSG), Multi-Scale Unique Color (MUC), and LLM-generated surprise scores. It evaluates these features on existing complexity benchmarks and introduces a new dataset, Surprising Visual Genome (SVG), with 200 images, human pairwise complexity ratings, and human surprise ratings. The authors report that adding their features improves mean Spearman correlations over the segmentation/class-count baseline on selected datasets, and that the final combined model is competitive with a supervised CNN while remaining interpretable.","tokens_in":18746,"tokens_out":7625,"duration_ms":70216,"significance":"If the quantitative claims survive a label-independent evaluation protocol, the paper makes a useful contribution: it identifies concrete structural, chromatic, and semantic dimensions beyond segment/class counts; introduces the SVG dataset with human ratings; validates LLM surprise scores against human surprise judgments (ρ = 0.73); and provides code, data, and detailed permutation-based ablations. The interpretability goal is well served by the algorithmic definitions and qualitative examples. The main weakness is that several feature-selection decisions use the evaluation labels, so the reported effect sizes are optimistic and need re-estimation under a fixed protocol.","major_comments":[{"comment":"The headline results are based on post-hoc dataset selection. The Datasets section states: 'We select image subsets to demonstrate the effectiveness of our new features to explain complexity - VISC and IC9600 architecture subset for MSG, Savoias Art and Suprematism subsets for MUC, and Savoias Interior Design and IC9600 Abstract for the combination of MSG and MUC.' Because these subsets were chosen after observing which datasets benefit, the increments in Table 1 (e.g., +0.13 on VISC, +0.10 on IC9 Architecture, +0.26 on Sav. Interior Design) are selected maxima, not expected gains for a randomly chosen dataset. The central claim that the two-feature model is insufficient should be supported by per-feature results on the complete set of datasets (Appendix Table 2 reports only the final model), or by a fixed, pre-specified dataset selection rule with multiple-comparison correction.","section":"Datasets; Table 1"},{"comment":"Several hyperparameters are tuned against the same ground-truth labels used for evaluation. The MUC section reports 'we report results using the best bit precision for each dataset in terms of correlation to complexity,' so the MUC gains (e.g., +0.08 on Sav. Art, +0.05 on Sav. Suprematism) are maxima over b in {3,...,8}. In addition, the color-versus-grayscale choice for MSG is based on permutation tests against complexity on the same datasets, the MSG/MUC scale weights are selected from the ablations in Appendix Table 4, and the square-root transform was retained because it 'achieved slightly better results' than log. None of these choices is nested inside the 3-fold cross-validation, so the reported Spearman correlations are optimistically biased. A fixed-protocol rerun (e.g., fixed b, or selection inside training folds only) is needed before the magnitudes of the improvements can be interpreted.","section":"Multi-Scale Unique Color; Multi-Scale Sobel Gradient; Appendix"},{"comment":"The choice of Gemini-1.5-Flash as the surprise scorer is also made on the basis of the evaluation labels: Appendix Table 3 compares three LLMs by the Spearman correlation of the resulting surprise features with SVG ground-truth complexity, and Gemini is selected because it performs best. The reported +0.05 surprise improvement on SVG is therefore not an unbiased estimate for a fixed, pre-specified LLM. Please report the SVG result for a pre-specified LLM or select the LLM on a held-out portion of the data, and provide a confidence interval or permutation test for the +0.05 increment.","section":"Appendix: Ablations on LLMs for Surprise Scores"},{"comment":"No uncertainty quantification is provided for the headline correlation coefficients or increments. With dataset sizes as small as 200 (SVG) and 49 (RSIVL), differences of 0.03-0.05 in Spearman correlation may be within sampling noise. Please report bootstrap or permutation-based confidence intervals for the Table 1 increments, or at least significance tests for each increment.","section":"Results; Table 1"},{"comment":"The instruction text in Figure 7, which is supposed to describe the visual complexity experiment, says 'simply click on the image you judge to be more surprising.' If participants actually saw these instructions, the SVG complexity ratings are contaminated by surprise, which would make the surprise-feature result partly circular. Please clarify whether this is a typographical error in the figure and confirm the exact wording shown to participants.","section":"Appendix Figure 7"}],"minor_comments":[{"comment":"The text says 'A histogram of these scores shows clear separation' but no histogram appears in the main text; please add the figure or move it to the appendix with a reference.","section":"SVG: A Dataset of Surprising Images"},{"comment":"The table caption does not define the '↑' notation used in the rows for 'Baseline + visual features'; please define it and state what test, if any, supports the arrowed increments.","section":"Table 1 caption"},{"comment":"The figure caption does not mention the red highlighting used for surprise-related words; please add a note explaining the highlighting.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a vision/cognition journal, and the released code and data are valuable. My main reservation is the in-sample feature selection: if the authors cannot supply a label-independent selection protocol or a rerun with fixed hyperparameters, the central quantitative claim would need to be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This paper does real empirical legwork: it builds SVG, a new human-rated dataset of surprising images, validates LLM surprise scores against human surprise ratings, releases code and data, and convincingly shows through examples and ablations that Shen et al.'s segmentation-plus-class-count model misses structural, chromatic, and semantic components of perceived complexity. The failure-mode images, the participant strategy quotes, and the honest limitation statements about SVG's manual construction are genuine contributions. The general conclusion that complexity is multifaceted and partly dataset-specific is plausible and likely correct. But the quantitative support for the headline increments is weaker than the tables imply. The MUC bit precision is chosen per dataset \"in terms of correlation to complexity\" (Methods, Multi-Scale Unique Color). The dataset subsets for MSG and MUC are explicitly \"selected to demonstrate the effectiveness\" of the new features (Datasets section). The surprise scorer was chosen after comparing three LLMs on the same SVG complexity labels (Appendix Table 3). Each choice is small, but together they make the reported gains (e.g., +0.08 on Sav. Art, +0.26 on Sav. Int. Design) selected maxima, not unbiased estimates. The stress-test note lands on reading the paper; this is a real flaw in the evaluation protocol. The paper is also honest that its ablation on weights and scales is preliminary, which is good, but it does not change the selection problem. The central qualitative claim survives, but the magnitudes should be treated as provisional. A fixed, label-independent feature-selection protocol, ideally pre-registered or with a held-out validation split, plus confidence intervals on the Spearman correlations, would make the paper much stronger. This is a paper for the visual complexity community and for anyone studying interpretable image feature models. It deserves serious refereeing — the dataset, the surprise dimension, and the failure-mode analysis are worth engaging with — but the quantitative claims will need tightening before the specific numbers should be trusted. My recommendation: send it to peer review with a clear request for a rerun under a fixed protocol.","headline":"A useful, well-documented empirical challenge to the simple two-feature account of visual complexity, but the headline gains are selected maxima rather than unbiased estimates because the evaluation uses the same labels for feature and hyperparameter choices.","tokens_in":19313,"tokens_out":2081,"would_cite":true,"duration_ms":21990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that visual complexity predictions improve when structural regularity, color diversity, and semantic surprise are added to segmentation and class counts, and that the combined interpretable model stays competitive with a…","keywords":["visual complexity","Multi-Scale Sobel Gradient","Multi-Scale Unique Color","colorfulness","surprise","large language models","image segmentation","SVG dataset"],"falsifier":"Run the same regressions on the Savoias Art and Suprematism subsets with MUC computed at a fixed bit precision chosen before seeing the complexity labels (for example, 7 bits for every dataset) and check whether the reported +0.08 and +0.05 correlation gains over the segment-and-class baseline survive; if they vanish, the part of the central claim that MUC adds genuine predictive value would fail.","tokens_in":18256,"feed_emoji":"🧩","tokens_out":11447,"duration_ms":94399,"temperature":0.7,"pith_summary":"This paper argues that a recent interpretable baseline—predicting perceived complexity from the square roots of segment and semantic-class counts—under-explains human judgments on several image domains. To fill the gap, it introduces two pixel-level features and one semantic feature: Multi-Scale Sobel Gradient (MSG), Multi-Scale Unique Color (MUC), and LLM-generated surprise scores, plus a new dataset of surprising images with human complexity ratings. The paper reports that each feature adds variance on datasets where the baseline fails, and that the combined model remains competitive with a supervised deep network while preserving interpretability. A sympathetic reader should care because this challenges the idea that visual complexity is simply object and segment numerosity, and it offers transparent features usable wherever complexity estimates must be explainable.","feed_headline":"Structure, color, and surprise each move perceived complexity","feed_subtitle":"Structure, color, and surprise beat the segment-count model on six datasets.","key_machinery":"The argument is carried by three features stacked onto the two-feature segment-and-class baseline. MSG (Multi-Scale Sobel Gradient) applies the Sobel operator at four scales over RGB channels and averages the gradient magnitudes; symmetric patches cancel, so MSG acts as a local symmetry detector and quantifies structural regularity. MUC (Multi-Scale Unique Color) quantizes each RGB channel to a lower bit precision, counts unique color triples after flattening, and repeats this at four spatial scales with fixed weights; it measures color diversity at multiple resolutions. Surprise is a 0–100 rating elicited from an LLM with zero-shot chain-of-thought prompting, yielding both a number and a natural-language explanation. A linear model with the square-root segment and class counts augmented by these features is evaluated by cross-validated Spearman correlation, with permutation tests used to decide which feature wins on which dataset.","core_discovery":"The central claim is that a two-feature account of visual complexity—regressing perceived complexity on the square roots of the number of image segments and semantic classes—fails precisely where segment and class counts are uninformative, and that three additional explainable features repair the account. Multi-Scale Sobel Gradient (MSG), a multi-resolution gradient statistic over RGB channels, captures local asymmetry and texture regularity; Multi-Scale Unique Color (MUC), a per-channel bit-quantized count of unique colors at four spatial scales, captures color diversity; and an LLM-generated surprise score captures whole-image semantic novelty. On the datasets where the baseline struggles, these features add variance: MSG raises the reported Spearman correlation on VISC from 0.56 to 0.68 and on the architecture subset of IC9600 from 0.66 to 0.76; MUC raises the Savoias Art subset from 0.73 to 0.81 and the Suprematism subset from 0.89 to 0.94; MSG plus MUC lifts the Interior Design subset from 0.61 to 0.87 and the Abstract subset from 0.66 to 0.83; and surprise raises the new SVG dataset from 0.78 to 0.83. The paper concludes that the combined interpretable model is competitive with a supervised deep network while requiring no training, and that visual complexity perception is genuinely multi-factorial.","pith_inferences":["Beyond the paper: the LLM-generated explanations could be parsed into object-level surprise (unusual subject, unusual activity, unusual object-context pairing), turning the holistic score into a decomposition testable against the scene-graph annotations inherited from Visual Genome.","Beyond the paper: coupling the paper's reported weak surprise–memorability correlations with a paired memory experiment—surprising vs. ordinary images matched on segment and class counts—would test whether surprise is a common cause of both complexity and memorability.","Beyond the paper: because MSG and MUC each help most where images are regular or color-dominant, applying the same two features to GUI or information-visualization complexity could reveal domain-specific weights, a practical extension of the paper's dataset-aware finding."],"forward_implications":["If the central claim holds, the two-feature segmentation model is not a complete account; models of perceived complexity should include structural regularity and color diversity as separate explanatory variables.","The gains concentrate where segment and class counts fail, so complexity models may need to be dataset- or domain-aware rather than universal across image categories.","LLM-generated surprise is a viable interpretable semantic feature: it adds unique variance on the SVG dataset and matches a supervised model's correlation, and the accompanying natural-language explanations make the prediction auditable.","An interpretable combination of these features can approach supervised accuracy without being trained on a complexity dataset, which would help applications requiring transparent complexity estimates."],"supporting_citations":[{"why":"Supplies the two-feature segment-and-class baseline that all comparisons extend.","marker":"(Shen et al., 2024)"},{"why":"Defines the colorfulness concept from which the Multi-Scale Unique Color feature is derived.","marker":"(Teresa et al., 2014)"},{"why":"Provides the zero-shot chain-of-thought prompting method used to elicit the LLM surprise scores.","marker":"(Wei et al., 2023)"},{"why":"Is the large language model whose outputs generate the surprise ratings and explanations.","marker":"(Gemini Team, 2024)"},{"why":"Provides the Visual Genome image pool and annotations used to build the SVG dataset.","marker":"(Krishna et al., 2016)"},{"why":"Supplies the IC9600 complexity scores used for stratification and the supervised benchmark the final model is compared against.","marker":"(Feng et al., 2023)"},{"why":"Provides the RSIVL dataset and the handcrafted-feature baseline.","marker":"(Corchs et al., 2016)"},{"why":"Provides the VISC dataset and the patch-symmetry structural feature that MSG is tested against.","marker":"(Kyle-Davidson et al., 2023)"},{"why":"Supplies the pairwise-comparison sampling and rating-rescaling procedure used to collect SVG complexity ratings.","marker":"(Saraee et al., 2018)"},{"why":"Provides the rank-analysis method that converts pairwise comparisons into scalar complexity scores.","marker":"(Bradley & Terry, 1952)"}],"fun_headline_variants":["Gradient, color, surprise decode visual complexity","Beyond segment counts: what really drives complexity","Three features reveal why complexity isn't simple","Explainable cues beat deep nets on complexity","Structure, color, surprise: keys to perceived complexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that selecting the MUC bit precision per dataset after looking at the human complexity ratings does not inflate the reported correlations; if that in-sample tuning is effectively overfitting, the gains attributed to MUC would be smaller than claimed.","fun_headline_variants_meta":{"raw":{"variants":["Gradient, color, surprise decode visual complexity","Beyond segment counts: what really drives complexity","Three features reveal why complexity isn't simple","Explainable cues beat deep nets on complexity","Structure, color, surprise: keys to perceived complexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1500,"prompt_tokens":1085,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":701,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":701,"tokens_out":415,"duration_ms":4439,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:49:22.058800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same regressions on the Savoias Art and Suprematism subsets with MUC computed at a fixed bit precision chosen before seeing the complexity labels (for example, 7 bits for every dataset) and check whether the reported +0.08 and +0.05 correlation gains over the segment-and-class baseline survive; if they vanish, the part of the central claim that MUC adds genuine predictive value would fail.","supporting_citations":[{"cited_title":", Ciocca, G","cited_arxiv_id":null,"evidence_quote":"Defines the colorfulness concept from which the Multi-Scale Unique Color feature is derived."},{"cited_title":", Ciocca, G","cited_arxiv_id":null,"evidence_quote":"Provides the RSIVL dataset and the handcrafted-feature baseline."}],"review_version":1}