{"id":"c5b33a0c-a90b-4162-b6ae-f42ddca20899","arxiv_id":"2501.15144","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning vision-language models with sentence-formatted outputs instead of tuple outputs improves shape attribute and coordinate prediction for larger models, and scaling the loss on numeric tokens sharpens numerical accuracy.","lead":"This paper tests whether AI models that describe images in full sentences learn spatial measurements better than models that output structured tuples. The authors built a synthetic 2D-shape benchmark and found sentence-style outputs help larger models, especially on new image arrangements, and that boosting numeric tokens during training improves coordinate guesses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sentence-vs-tuple advantage may be an artifact of the regex/JV evaluation pipeline rather than a difference in learned ability; the paper's own Limitations admit regex patterns perform poorly on unstructured outputs.","rationale":"The reader's weakest_assumption already identifies the parser/evaluation-pipeline concern, so I partially agree. However, I elevate it to the single most load-bearing issue: the benchmark and fine-tuning are sound, but every format-comparison result in Tables 2, 3, and 4 passes through regex and JV matching that is not shown to be format-neutral. The paper's own Limitations flag exactly this vulnerability. The proposed test — an LLM-based parser as a control — would directly decouple format-driven learning from format-driven evaluation. I also note the secondary issue of missing repeated runs and confidence intervals, which the reader raised; but the parser confound alone is enough to make the headline claim conditional pending verification. The verdict should remain CONDITIONAL rather than ACCEPT because the central claim's validity depends on a check that is not currently reported.","tokens_in":20621,"tokens_out":4067,"duration_ms":40804,"concrete_test":"Recompute SAMA on the saved MiniCPM-V2.6 sentence-format and tuple-format predictions (or regenerate them) for all OD test sets, replacing the regex/JV pipeline with an LLM-based parser that converts each raw prediction into a canonical JSON structure before attribute matching. If the sentence-vs-tuple SAMA gap persists under this format-agnostic extraction, the format claim is unlikely to be a parsing artifact; if the gap shrinks or reverses, the reported advantage is an evaluation artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that sentence-formatted outputs outperform tuple formats rests on SAMA, a metric that assumes outputs are properly structured for regex extraction and Jonker-Volgenant edit-distance matching (Section 3.4). The Limitations explicitly state: \"Regex patterns perform poorly for unstructured predictions.\" Both sentence and tuple formats can deviate from the exact template under model error, but the two formats fail in different ways: tuple outputs are rigid key=value lists and a single malformed token (e.g., a missing bracket or a space in 'center_coordinates') can make the whole entry unparseable, while sentence outputs are closer to natural language and may be partially recoverable by regex even when imperfect. If the regex patterns and edit-distance matching happen to be better tuned for the sentence template, the reported SAMA/RMSE advantage could reflect parser robustness rather than visual understanding. This is not a hypothetical: Table 2 shows small models with tiny absolute differences (Qwen-2B: 0.048 vs 0.054 on OD Composition) and Table 3 shows large RMSE swings; without a parser-invariant check, the format comparison is confounded. The same pipeline is applied to single 200-sample runs with no confidence intervals, so chance cannot be excluded, but the parser confound is the more systematic threat because it biases the central comparison in a single direction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a synthetic benchmark of 2D primitive shapes with varied spatial attributes, fine-tunes six open-source VLMs (2B-8B) with LoRA, and studies two output formats (sentence vs. tuple) plus a numeric-token loss-scaling technique. It proposes a custom metric, SAMA, based on Jonker-Volgenant matching and regex-based attribute extraction, and validates the approach on a plant-phenotyping dataset. The headline claims are that sentence-formatted targets outperform tuple-formatted targets, especially under large out-of-domain shifts, and that scaling the loss on numeric tokens improves numerical approximation. The code is publicly released.","tokens_in":20912,"tokens_out":4961,"duration_ms":43325,"significance":"If the claims held, the paper would provide a simple, practical recipe for fine-tuning VLMs on spatial measurement tasks: use sentence-formatted targets and weight numeric-token losses. The benchmark itself, with controlled OD scenarios and multiple model sizes, is a useful contribution, and the inclusion of a real-world plant-phenotyping transfer experiment strengthens the empirical scope. The paper is also generally reproducible in structure: hyperparameters, LoRA ranks, and dataset construction are described in detail, and the code is available. However, the headline claims are broader than the reported data support, and the main comparison is vulnerable to a parser confound and to the absence of uncertainty estimates, so the significance of the conclusions is currently limited.","major_comments":[{"comment":"The claim that 'models fine-tuned on the sentence format consistently outperform their tuple-based counterparts' and the abstract's statement that 'sentence-based outputs outperform tuple formats' are contradicted by the paper's own data for smaller models. In Table 2, Qwen-VL 2B achieves higher SAMA in tuple format on OD Occlusion (0.013 vs. 0.004), OD Rotation (0.007 vs. 0.002), Test Set (0.010 vs. 0.002), and OD Size (0.008 vs. 0.003); Paligemma achieves higher tuple SAMA on five of the six columns. In Table 3, Qwen-VL 2B has lower center RMSE in tuple format on all six columns (e.g., 81.604 vs. 90.314 on OD Composition), and in Table 6, Paligemma and Qwen-VL 2B have consistently higher F1 in tuple format. The claims should be restricted to the 7B-8B models and to specific metrics, and the main text should explicitly acknowledge the counter-examples.","section":"Section 4.2, Tables 2, 3, and 6"},{"comment":"The central sentence-vs-tuple comparison is potentially confounded by the evaluation pipeline. SAMA assumes that outputs are structured enough for regex extraction and Jonker-Volgenant edit-distance matching, and the Limitations state that 'Regex patterns perform poorly for unstructured predictions.' Tuple and sentence formats fail in different ways: a single malformed token in a tuple can make an entire entry unparseable, while a sentence output may be partially recoverable even when it deviates from the template. Since both formats are scored by the same regex/JV machinery, the reported SAMA/RMSE advantage may reflect parser robustness rather than visual understanding. A parser-invariant check is needed, such as evaluating both formats with an LLM-based extractor or restricting the comparison to outputs that are parseable under both regex schemas.","section":"Section 3.4 and Section 7 (Limitations)"},{"comment":"All quantitative conclusions rest on single runs with no confidence intervals, and the five OD test sets contain only 200 samples each. Many of the differences used to support the format claim are very small (e.g., Table 2: Qwen-VL 2B SAMA 0.048 vs. 0.054 on OD Composition; Phi-V 0.027 vs. 0.024). Without repeated seeds, error bars, or a significance test, these differences are within the range of run-to-run noise. At minimum, the paper should report the number of seeds and standard deviations, or explicitly temper claims that are based on such small absolute differences.","section":"Section 3.2 and Section 4.1"},{"comment":"The loss-scaling claim is weakened by a selection effect and by inconsistent results. The scales 1.5, 2, 2.5, and 3.5 are evaluated on the same OD test sets that are used to report the final results, so the reported 'best' scale is selected on the test data rather than on a held-out validation set. Moreover, the improvement is not consistent: in Table 5, SW 2.0 improves center RMSE on OD Composition (31.81 vs. 34.05 for sentence baseline) but worsens it on OD Size (82.39 vs. 80.66), and Table S2 shows that scaling generally reduces SAMA accuracy. The paper should use a held-out validation split for scale selection and should quantify the accuracy trade-off before claiming that scaling 'enhances numerical approximation capabilities.'","section":"Section 4.2, Table 5, and Table S2"},{"comment":"The discussion and conclusion overstate the generality of the findings. Section 5 says 'larger models fine-tuned in the sentence format outperform those fine-tuned in the tuple format by a wider margin compared to smaller models,' but even for MiniCPM-V2.6, Table 6 shows that tuple format yields higher or equal F1 on most OD sets (e.g., 0.742 vs. 0.733 on OD Composition, 0.804 vs. 0.814 on OD Occlusion, 0.809 vs. 0.810 on OD Rotation). The conclusion's statement that 'sentence-based formats consistently enhancing both numerical and spatial tasks' is therefore not supported by the full set of reported tables.","section":"Section 5 and Section 6"}],"minor_comments":[{"comment":"The row label 'Ph3i-V' appears to be a typo for 'Phi3-V'; it should be consistent with Table 6.","section":"Table 3"},{"comment":"The column header 'OD Test Set' is ambiguous: it is not one of the five OD sets listed in Table 1 and appears to denote the in-domain test set. It should be labelled 'Test Set' consistently with Tables 2 and 3.","section":"Table 5"},{"comment":"The description of SAMA would benefit from explicit details about the regex patterns used for each attribute and about how edit-distance cost is computed across attributes of different types (discrete vs. continuous). Without these details, the metric is hard to reproduce independently.","section":"Section 3.4"},{"comment":"Table S4 shows that sentence format is not consistently better on center-coordinate RMSE for the plant dataset (e.g., 152.31 vs. 121.78 on the train split), and the appendix attributes the discrepancy to linear matching. This should be acknowledged in the main text when citing Table 4, since the main text currently highlights only the count RMSE advantage.","section":"Appendix A.2.2 and Table S4"},{"comment":"The qualitative outputs for Qwen-2B, Qwen-7B, Phi-3.5, and Paligemma contain many garbled tokens (e.g., 'Inoendo|1|>', 'Tquare', 'notcluded'). These figures are difficult to interpret and should either be replaced by cleaner representative outputs or annotated to show what is being compared.","section":"Figures S5-S8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer-vision applications venue and the benchmark is potentially reusable, but the abstract and conclusion currently overclaim relative to the reported tables. The strongest revision path is to narrow the claims to the models and metrics that support them, add uncertainty estimates, address the parser-confound concern, and re-run loss-scale selection on a held-out split. I would not reject the paper, but the central claims need substantive revision before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper builds a clean synthetic benchmark for VLM spatial measurement, compares sentence versus tuple output formats under LoRA fine-tuning, and tries a numeric-token loss scaling trick. That is a legitimate, useful empirical contribution, and the code is out. The real-world plant phenotyping check is a nice touch. I think the work deserves a serious referee, but the central claim needs to be pulled back and the evaluation pipeline needs a robustness check.\n\nWhat is actually new: the benchmark with five controlled out-of-domain shifts (composition, occlusion, rotation, size, spatial layout) is a reasonable testbed, and SAMA—the Jonker-Volgenant edit-distance matching metric for compositional outputs—is a sensible attempt to score shape-attribute assignments. The comparison of sentence versus tuple formats and the loss scaling on numeric tokens are legitimately new experiments, not just a retread of prior work.\n\nWhere the soft spots are, in proportion: the abstract says sentence-based outputs outperform tuple formats, but the paper's own tables show that for Qwen-2B and Paligemma, tuple is often better on F1 and sometimes on center RMSE (Tables 3 and 6). The consistent sentence advantage is mainly in the larger MiniCPM and Qwen-7B models. So the headline is broader than the data. There are no error bars or repeated runs; OD sets have only 200 samples, so small differences are within noise. The numeric-token loss scale was chosen by evaluating 1.5, 2, 2.5, 3.5 on the reported test sets, which adds a selection effect—the gains in Table 5 are partly a result of peeking at the test set.\n\nThe stress-test note about the parser is not hypothetical. The Limitations admit: \"Regex patterns perform poorly for unstructured predictions.\" Sentence and tuple outputs fail in different ways under model error: a single malformed token in a tuple can make the whole entry unparseable, while a sentence may still be partially recoverable. So the reported format advantage could reflect parser robustness rather than visual understanding. The right fix is a parser-invariant check—e.g., parse both formats with an LLM-based extractor, or score raw text directly. Without that, the main comparison is confounded.\n\nThis is not a fatal flaw. The larger-model results, the loss-scaling effect, and the benchmark itself are all worth having. But the paper needs a major revision: qualified claims, error bars, held-out scale selection, and a parser-robust evaluation. I would send it to peer review, and I would want the authors to address the parser concern before acceptance. For a reader working on spatial reasoning in VLMs, the benchmark and the qualified recipe are useful; for the broader VLM community, it is a solid but incremental contribution.","headline":"A useful new benchmark and a plausible training recipe, but the sentence-vs-tuple headline is overstated and the main comparison may be partly an artifact of the evaluation parser.","tokens_in":21390,"tokens_out":1836,"would_cite":true,"duration_ms":19970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that how a vision-language model is told to write its answer changes how well it measures what it sees, with sentence-formatted targets beating tuple-formatted ones and numeric-token loss scaling further improving…","keywords":["vision-language models","out-of-domain generalization","output format","spatial measurement","loss scaling","primitive shapes","structured attribute matching accuracy","fine-tuning"],"falsifier":"Run the sentence-versus-tuple comparison with a format-agnostic scorer that converts both output formats into the same structured object before matching, and check whether the sentence advantage in SAMA and RMSE survives; if it shrinks or reverses, the claim that sentence format improves learning would not be supported.","tokens_in":20442,"feed_emoji":"📐","tokens_out":8950,"duration_ms":77117,"temperature":0.7,"pith_summary":"This paper asks whether the way a vision-language model is told to write its answer changes how well it can measure what it sees. By fine-tuning open-source vision-language models of 2B to 8B parameters on a controlled benchmark of simple 2D shapes, the authors try to show that coherent sentence-style outputs produce better spatial and numerical predictions than compact tuple-style outputs, especially when test conditions move far from training. They also claim that scaling the loss on numeric tokens during fine-tuning improves coordinate and rotation estimation. The paper introduces its own benchmark and a matching-based accuracy metric, and validates the format effect on a real plant-phenotyping dataset under domain shift. If these claims hold, output-format design and loss weighting become cheap, concrete levers for improving fine-grained spatial reasoning in vision-language models.","feed_headline":"Sentence outputs beat tuples for VLM spatial measurement","feed_subtitle":"Natural-language targets beat structured tuples for fine-tuning VLMs to measure shapes.","key_machinery":"The argument is carried by four coupled devices. The benchmark generates images from known configurations of shapes, colors, quadrants, center coordinates, rotations, occlusions, and relative positions, so every prediction has a ground truth. The two output formats, sentence and tuple, encode the same ground truth differently, isolating the effect of surface format on learning. The evaluation metric, Structured Attribute Matching Accuracy (SAMA), uses the Jonker-Volgenant algorithm to solve the linear assignment between predicted and ground-truth shape-attribute sets, with edit distance as the cost, then reports per-attribute accuracy and RMSE for continuous attributes. The fourth device is loss scaling: multiplying the cross-entropy loss on predefined numeric tokens during fine-tuning, which the paper shows improves numeric approximation.","core_discovery":"The paper's central claim is that output format is a learning signal for vision-language models, not just a reporting choice. On a synthetic benchmark of 2D shapes with controlled attributes, models fine-tuned on full-sentence descriptions of each shape consistently beat models fine-tuned on tuple-formatted outputs carrying the same information, and the advantage grows on out-of-domain test sets with larger domain gaps. The paper further claims that scaling the cross-entropy loss on numeric tokens by a factor of roughly 2 to 2.5 improves numeric approximation, lowering coordinate error further at a small accuracy cost. A real-world plant-phenotyping experiment supports the format effect: under the hardest distribution shift, leaf-count error is substantially lower for sentence-format fine-tuning than for tuple-format fine-tuning. The paper also introduces SAMA, a Jonker-Volgenant assignment-based metric, to score shape-attribute predictions when predicted shapes appear in arbitrary order.","pith_inferences":["The authors do not test this, but the format effect suggests that aligning output structure with the LLM's pretraining distribution is a general training design choice, applicable to bounding-box regression, depth estimation, and other continuous-output VLM tasks.","A testable extension: paraphrase the same sentence template to check whether the advantage comes from surface form or from coherent sentence structure.","Another extension: combine sentence format with numeric-token loss scaling on larger models and measure whether the gains are additive.","The synthetic benchmark could be pushed with unseen colors, shape combinations, or perspective distortion to see where the format advantage saturates."],"forward_implications":["Sentence-formatted targets become the safer default for fine-tuning VLMs on structured spatial tasks, since the paper's experiments show they improve both attribute-matching accuracy and coordinate and rotation error across out-of-domain sets.","Weighting numeric tokens in the loss by a factor of 2 to 2.5 is a cheap, complementary intervention that lowers coordinate RMSE, with a small accuracy trade-off that can be tuned.","The format advantage widens as distribution shift increases, so output-format choice matters most exactly when generalization is hardest.","Larger models gain more from sentence format than smaller models, which can detect attributes but often misassign them; format interventions alone are unlikely to fix small-model spatial grouping.","SAMA offers a reusable way to score compositional outputs with unordered shape-attribute sets, which could transfer to other structured prediction benchmarks."],"supporting_citations":[{"why":"Supplies the Jonker-Volgenant assignment algorithm used by SAMA to match predicted and ground-truth shape-attribute sets.","marker":"[6]"},{"why":"Provides the linear-assignment matching idea that SAMA adapts for compositional shape-attribute evaluation.","marker":"[5]"},{"why":"Supplies the edit-distance algorithm used as the cost function in SAMA's matching.","marker":"[9]"},{"why":"Supplies the MiniCPM-V models and the fine-tuning settings, including LoRA, optimizer, and schedule, used across experiments.","marker":"[23]"},{"why":"Supplies the Qwen2-VL models evaluated as 2B and 7B baselines.","marker":"[21]"},{"why":"Supplies the real-world plant-phenotyping dataset used to validate the format effect under distribution shift.","marker":"[12]"},{"why":"Supplies the MD5 hashing used to guarantee unique benchmark images and configurations.","marker":"[16]"},{"why":"Supplies the Paligemma model evaluated as a 3B baseline.","marker":"[4]"},{"why":"Supplies the Phi-3 vision model evaluated as a 4.5B baseline.","marker":"[1]"}],"fun_headline_variants":["Sentences beat tuple labels for VLM shape measurement","Output format shapes VLM learning, not just reporting","Sentence targets aid VLM generalization to new domains","Numeric loss scaling boosts VLM coordinate accuracy","VLM fine-tuning: sentence format outshines tuples on spatial tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the assumption that the evaluation pipeline scores sentence and tuple outputs fairly, but SAMA extracts attributes with pattern matching that may parse one format more easily than the other.","fun_headline_variants_meta":{"raw":{"variants":["Sentences beat tuple labels for VLM shape measurement","Output format shapes VLM learning, not just reporting","Sentence targets aid VLM generalization to new domains","Numeric loss scaling boosts VLM coordinate accuracy","VLM fine-tuning: sentence format outshines tuples on spatial tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1667,"prompt_tokens":909,"completion_tokens":758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":525,"tokens_out":758,"duration_ms":7173,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:34:07.175161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the sentence-versus-tuple comparison with a format-agnostic scorer that converts both output formats into the same structured object before matching, and check whether the sentence advantage in SAMA and RMSE survives; if it shrinks or reverses, the claim that sentence format improves learning would not be supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Jonker-Volgenant assignment algorithm used by SAMA to match predicted and ground-truth shape-attribute sets."},{"cited_title":"Carion, F","cited_arxiv_id":null,"evidence_quote":"Provides the linear-assignment matching idea that SAMA adapts for compositional shape-attribute evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the edit-distance algorithm used as the cost function in SAMA's matching."},{"cited_title":"Minervini, A","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world plant-phenotyping dataset used to validate the format effect under distribution shift."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MD5 hashing used to guarantee unique benchmark images and configurations."}],"review_version":1}