{"id":"c0e2fbef-108d-43de-b41d-127c42fe0892","arxiv_id":"2412.16599","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new synthetic-image benchmark, CDR, shows that leading multimodal models mostly guess on compass direction questions, with chain-of-thought fine-tuning only partially closing the gap.","lead":"This paper introduces a benchmark for testing whether multimodal AI models understand compass directions like north and south, not just left and right in a picture. It finds most models perform near random guessing on compass questions, and that step-by-step fine-tuning helps only partially.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on an unverified 100% human baseline and unvalidated manual icon annotations; low model scores may reflect task ambiguity rather than missing compass-direction reasoning.","rationale":"I read the paper as primarily claiming that current MLMs do not understand compass direction (near random on CDR) and that mixdata+CoT fine-tuning substantially improves this on the icon variant. For the first claim to hold, CDR must be a valid instrument. The manuscript provides no human validation: no participants, no inter-annotator reliability, no pilot. The one explicit human number, 100% in Figure 1, is presented as a fact without a protocol. Given that the icon orientations are manually annotated from Internet images and the task quantizes continuous orientations into 8 categories, there is a real risk that label noise or ambiguity explains part of the poor performance. I agree with the reader's weakest_assumption. However, the letter and number variants are procedurally generated and unambiguous, and models also score near chance there (e.g., 13.04% and 13.54% for LLaVA-7B), which independently supports the 'models struggle' conclusion for those formats. Thus the concern is not a refutation but a blocker for treating the benchmark and the fine-tuning numbers as established. The Table III text/table mismatch (11.90% vs 10.69% for 'All') is a minor but real reporting inconsistency that should be fixed. On balance, the reader's CONDITIONAL verdict is appropriate; my read does not move it.","tokens_in":8188,"tokens_out":6734,"duration_ms":60293,"concrete_test":"Release the CDR test split and run a preregistered human baseline on a stratified random sample of 300 items (100 icon, 100 letter, 100 number) from the Relative Compass Reasoning tasks. Each item should be answered by at least 5 naive participants given exactly the model prompt, with no training. Compute accuracy and Fleiss' kappa. If human accuracy is >=95% and kappa >=0.8, the ambiguity objection is refuted and model scores are interpretable; if human accuracy is <90% or kappa <0.7, the benchmark's icon labels are unreliable and the central claim must be revisited. As a secondary check, recompute Table III's 'All' row from the released data to resolve the 11.90% vs 10.69% discrepancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that most MLMs perform at random-guessing levels on compass direction reasoning—depends on the CDR benchmark being an unambiguous measurement of that ability. Section II-A states that icon orientations are manually annotated with no inter-annotator agreement protocol, and the paper asserts human-level performance of 100% (Figure 1) without describing any human-subject experiment or baseline procedure. The icon images are sourced from the Internet and discretized into 8 compass bins; a finger, arrow, or car that points off a cardinal axis must be assigned to one bin, and the correctness of questions such as Table I's Relative Compass Reasoning examples hinges on that assignment. If icons are visually ambiguous or annotators disagree, the near-chance model accuracies (e.g., 11.90% for LLaVA-7B) would reflect label noise and task artifacts, not cognitive failure. The same issue threatens the headline fine-tuning result (53.43%), which is measured on the icon subset. The letter and number tasks are less subject to this concern, partially supporting the authors' conclusion, but the icon task drives both the abstract's 'random guessing' language and the main fine-tuning claim. A second, smaller inconsistency (Section III-D text says full CDR training gives 11.90%, while Table III lists 'All' as 10.69% and 11.90% as the 7B Base) reduces confidence in the reporting even if it does not change the overall direction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Compass Direction Reasoning (CDR) benchmark for evaluating multimodal language models on spatial and compass direction reasoning. CDR contains three image modalities (icons, letters, numbers) and seven question types spanning object classification, absolute spatial/compass direction, and relative spatial/compass reasoning. The authors evaluate six MLMs zero-shot and report that most perform near or below random guessing (12.5%) on the relative compass reasoning tasks; for example, LLaVA-7B achieves 11.90% on Relative Compass Reasoning (icon). They then fine-tune LLaVA-7B on CDR data, alone and mixed with out-of-domain data, and with chain-of-thought (CoT) answers, reporting that a combination of CoT and mixed data raises accuracy to 53.43%. The central claim is that current MLMs lack robust compass direction reasoning and that mixdata plus explicit step-by-step reasoning is a partial remedy.","tokens_in":8455,"tokens_out":4386,"duration_ms":38173,"significance":"If the CDR benchmark is valid and made publicly available, it addresses a genuine gap: existing spatial reasoning benchmarks mostly test relative positions within images rather than mapping those positions to real-world compass directions. The finding that six commercial and open models perform at or near chance on relative compass reasoning is a useful, falsifiable result, and the fine-tuning study provides an actionable intervention (CoT with mixed data) that improves accuracy from near chance to 53.43% on the icon subtask. The paper's strengths include a large synthetic dataset with balanced answer distributions, systematic task progression, and controlled fine-tuning comparisons. However, the significance is currently conditional on validating the benchmark's low-ambiguity assumption and on releasing the dataset; without human-level validation and public data, the empirical claims cannot be fully assessed or reused by the community.","major_comments":[{"comment":"The human baseline of 100% accuracy in Figure 1 is asserted without any description of an experimental protocol: no number of participants, task instructions, or reporting procedure is given. Because the central claim that MLMs perform 'at random guessing levels' is meaningful only if the benchmark questions are trivially unambiguous for humans, the paper must include a proper human-subject evaluation with several annotators, report their accuracy, and report inter-annotator agreement on the direction labels. Without this, the low model scores could partly reflect task ambiguity rather than missing compass-direction reasoning.","section":"Section II-A and Figure 1"},{"comment":"The icon images are manually annotated for orientation into eight compass bins, but the manuscript provides no ambiguity checks, no inter-annotator agreement, and no statistics about the angles of the icons relative to the bin boundaries. Many icons (e.g., a pointing finger, a car) may not point exactly along one of the eight directions, and the discretization could introduce label noise. This is load-bearing because the headline fine-tuning result (53.43%) is measured on the icon subset, and the near-chance accuracy of LLaVA-7B (11.90%) could be an artifact of noisy or ambiguous labels. The authors should report the distribution of annotated angles, an agreement measure, and ideally re-run the key evaluations after excluding icons whose orientation is ambiguous.","section":"Section II-A"},{"comment":"The CDR benchmark is the main contribution of the paper, yet the manuscript does not state whether the dataset, annotation files, fine-tuning code, or evaluation scripts will be released. For a benchmark paper, public availability is essential for reproducibility and community adoption. The authors should provide a clear data-release statement, including a URL or hosting plan, and specify any licensing restrictions (especially for the icon images sourced from Iconfont).","section":"Data availability"},{"comment":"There is a direct inconsistency between the text and Table III. The text says 'Using the all Relative Compass Reasoning training data in (71552 samples), the model only get 11.90% accuracy', but Table III lists '7B Base' as 11.90% and 'All' as 10.69%. The correct reading is that fine-tuning on the full CDR data yields no improvement over the base model (indeed a slight drop). This misreporting should be corrected, and the narrative around 'limited improvements' should be aligned with the actual numbers. In addition, the fine-tuning results appear to be single runs with no variance reported; please provide results over multiple seeds or at least state the number of runs and observed variance.","section":"Section III-D and Table III"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'wu et al.' should be 'Wu et al.', 'llava v1 5 mix665k.json' should be 'LLaVA-v1.5-mix665k.json', 'V A' spacing in LLaVA appears inconsistently, and the number-type example in Table I contains 'a a Cartesian coordinate system'. Please proofread the manuscript carefully.","section":"Throughout"},{"comment":"Figure 2 contains corrupted labels such as '/glyph1197ortheast' and '/glyph1197orth' that appear to be PDF-extraction artifacts. The figure should be regenerated with correct text rendering.","section":"Figure 2"},{"comment":"The column headers in Table III are ambiguous: '7B Base All' is a list of three separate conditions (7B Base, All, All+20K, etc.) but is presented as a single row label. Please make the table structure clearer, for example by separating the base model column from the fine-tuning conditions.","section":"Table III"},{"comment":"The claim that All+40K corresponds to an 'optimal mix ratio (approximately 2:1)' is not substantiated: 71,552 CDR samples plus 40,000 mixdata samples gives roughly 1.8:1, and the selection of 40K is empirically driven without a statistical test. Please clarify the ratio and discuss whether the difference from All+20K or All+60K is significant.","section":"Section III-D"},{"comment":"The conclusion states that CDR contains 'over 100K training and testing samples', but the statistics in Section II-B sum to roughly 50K+50K+71K training and 4K+4K+4K+5K testing for the different tasks; the total should be stated per task to avoid ambiguity.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is a benchmark contribution, and the lack of any dataset release statement is a serious problem for a journal that values reproducible benchmarks. I also note that the paper cites the authors' own MODE paper [14] in the introduction without an obvious connection to direction reasoning; this is not disqualifying but is worth checking. If the icon images are protected by Iconfont's license, the authors should state any redistribution restrictions explicitly, since the benchmark's utility depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look: this paper does something nobody has done cleanly before—benchmarking compass direction reasoning in multimodal models, as opposed to the usual relative spatial reasoning tasks. The task family (rotated coordinate systems, absolute and relative compass directions) is simple but effective, and the core empirical observation is believable: object classification is near-perfect, yet on relative compass reasoning most models sit at or near the 12.5% chance level. The letter and number variants rule out the worry that it's just an icon-labeling artifact, since those also sit near chance. That is a genuinely useful result for anyone building navigation or embodied agents.\n\nThe fine-tuning story is more incremental but still interesting: mixing out-of-domain data at a 2:1 ratio and adding hand-written CoT answers lifts LLaVA-7B from 11.90% to 53.43% on the icon task. The CoT teaching the explicit mapping rule is a reasonable mechanism, not a circularity, since the test split is held out.\n\nNow the soft spots. The human baseline of 100% is asserted in Figure 1 with no protocol—no sample size, no participants, no ambiguity checks. At minimum they should report how it was measured; otherwise the 'random vs. human' framing is unverifiable. Second, the icon orientations are manually annotated from internet images with no inter-annotator agreement. The letter/number tasks mitigate this, but the headline fine-tuning result is on icons, so label noise could inflate or deflate the improvement. Third, there's a clear inconsistency: Section III-D says training on All gives 11.90%, but Table III lists the 7B Base as 11.90% and All as 10.69%. Small, but it erodes trust. Fourth, the dataset is not released, which limits the benchmark's immediate utility. No error bars on the fine-tuning numbers, though temperature 0 and a fixed test set reduce variance.\n\nOverall, the central claim holds up in its broad strokes—models do fail at compass reasoning—but the paper needs a real human baseline, a dataset release, and a fix for the numbers before I'd treat the fine-tuning gains as established.\n\nRecommendation: send it to review. The benchmark idea is good, the finding is publishable, and a good referee can force the authors to close the validity gaps. My own verdict would be conditional acceptance after those fixes.","headline":"A genuinely new compass-direction benchmark with a striking near-chance finding, held back by unreleased data and a missing human baseline.","tokens_in":8953,"tokens_out":2596,"would_cite":false,"duration_ms":22263,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most multimodal language models answer compass-direction questions no better than random guessing, and the paper shows that fine-tuning with diverse data plus chain-of-thought reasoning lifts a small model to 53% accuracy.","keywords":["compass direction reasoning","multimodal language model","spatial reasoning benchmark","chain-of-thought fine-tuning","LLaVA","direction reasoning evaluation","visual question answering","benchmark dataset"],"falsifier":"Have a group of human annotators answer the same relative-compass questions and measure their agreement with the automatically generated labels; if agreement is far from perfect, the benchmark itself is ambiguous and the model scores cannot be interpreted as a pure measure of compass reasoning.","tokens_in":7995,"feed_emoji":"🧭","tokens_out":5648,"duration_ms":46068,"temperature":0.7,"pith_summary":"The paper introduces the Compass Direction Reasoning (CDR) benchmark, a set of synthetic grid images and multiple-choice questions that test whether multimodal language models can map visual orientation to spatial (up/down/left/right) and compass (north/south/east/west) directions. Across six models, object recognition is strong but direction reasoning is weak: on relative compass tasks most models score near the 12.5% random-guessing baseline, far below the paper's assumed 100% human reference. The paper then shows that fine-tuning LLaVA-7B directly on CDR alone does not teach the rule, but mixing out-of-domain data with roughly a 2:1 ratio and adding chain-of-thought reasoning answers raises relative compass accuracy from 11.90% to 53.43%. The upshot is a demonstrated gap: current models have not internalized real-world compass rules, and step-by-step training with diverse data is a partial remedy.","feed_headline":"AI models mostly guess on compass-direction tests","feed_subtitle":"New CDR benchmark puts six multimodal models near chance; mixdata + CoT fine-tuning lifts LLaVA-7B to 53 percent.","key_machinery":"The load-bearing object is the CDR benchmark's question taxonomy, especially the Relative Compass Reasoning task. Images are 200x200 canvases with a 3x3 grid; icon, letter, and number stimuli place one central element and surrounding elements, and questions specify a coordinate convention (e.g., 'upward is West') or the central element's heading to force the model to rotate image-relative positions into compass labels. The mapping rule the paper exploits is the rotation between image axes and compass axes: once the model knows the transform from one spatial direction to one compass direction, all eight directions follow. Chain-of-thought fine-tuning works, in the paper's account, because it makes the model verbalize this rotation step-by-step instead of guessing the multiple-choice label directly.","core_discovery":"The paper's central claim is that multimodal language models can classify icons and describe spatial layout while failing to reason about compass directions that require mapping image-relative orientation onto geographic labels. On the Relative Compass Reasoning (icon) task, every evaluated model performs near or below random guessing, with GPT-4o-mini at 18.65% and the best, Claude-3-Sonnet, at 19.46%. Directly fine-tuning LLaVA-7B on 71,552 CDR relative-compass samples yields only 11.90%, showing that memorizing question-answer pairs is not enough; the model must acquire an abstract orientation rule. Adding 40,000 random out-of-domain samples lifts accuracy to 31.05%, and replacing the CDR portion with chain-of-thought answers that spell out the mapping from image directions to compass directions lifts it further to 53.43% when combined with the same 40K mix. The paper reads this as evidence that explicit step-by-step reasoning about orientation relationships, rather than more in-domain data, is what lets a smaller model begin to master compass reasoning.","pith_inferences":["Going beyond the paper, a natural next test is whether the 53.43% result transfers to real photographs or map layouts; grid-based icons may make the learned rule easier to verbalize than continuous real-world scenes.","Going beyond the paper, the paper's implied curriculum predicts that models with stronger spatial reasoning should improve more on CDR after chain-of-thought training, a correlation that could be tested across model families.","Going beyond the paper, if the 2:1 mix ratio is the real mechanism, the dominant factor might be regularization rather than world knowledge; an ablation holding total data fixed while varying the out-of-domain fraction would separate those accounts."],"forward_implications":["Compass direction reasoning is not captured by existing spatial-reasoning benchmarks: high spatial accuracy does not transfer to compass accuracy.","Training on a large in-domain direction dataset alone can leave a model at chance, so accuracy on direction tasks is not simply a data-volume problem.","A mix of roughly two parts in-domain data to one part out-of-domain data gives the best generalization among the mixtures tested, with larger mixes degrading performance.","Chain-of-thought supervision that verbalizes the spatial-to-compass mapping is a more effective training signal than the same questions with single-label answers.","State-of-the-art closed and open models tested all fall far below human-level compass reasoning, so real-world navigation and orientation tasks remain open problems."],"supporting_citations":[{"why":"Supplies the LLaVA-7B and LLaVA-13B model family that is evaluated and then fine-tuned.","marker":"[23]"},{"why":"Supplies the Claude-3-Haiku and Claude-3-Sonnet models whose near-chance CDR scores anchor the central failure claim.","marker":"[24]"},{"why":"Supplies GPT-4o-mini, one of the stronger spatial models whose compass accuracy still collapses to near chance.","marker":"[25]"},{"why":"Supplies Gemini-1.5-Pro, whose relative compass scores are among the outputs used to conclude current models lack compass reasoning.","marker":"[26]"},{"why":"Prior result that vision-language models struggle with spatial reasoning; CDR extends that finding to compass reasoning and motivates the benchmark.","marker":"[22]"},{"why":"Source of the high-resolution icon images used to build the icon-type CDR questions.","marker":"[27]"},{"why":"Visual instruction tuning method and the LLaVA model line that the fine-tuning experiments build on.","marker":"[13]"}],"fun_headline_variants":["MLMs flunk compass-direction reasoning: near chance","Compass reasoning stumps multimodal models","New benchmark: AI can't tell north from south in images","Step-by-step reasoning lifts compass accuracy to 53%","Why large vision-language models fail compass tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on the assumption that the synthetic grid questions are an unambiguous measure of compass-direction reasoning; the paper does not report human agreement or ambiguity checks, so if the rotated-coordinate wording or icon orientations are genuinely confusing, the low model scores would be artifacts rather than evidence of missing direction understanding.","fun_headline_variants_meta":{"raw":{"variants":["MLMs flunk compass-direction reasoning: near chance","Compass reasoning stumps multimodal models","New benchmark: AI can't tell north from south in images","Step-by-step reasoning lifts compass accuracy to 53%","Why large vision-language models fail compass tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1311,"prompt_tokens":932,"completion_tokens":379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":548,"tokens_out":379,"duration_ms":3992,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:25:58.760292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a group of human annotators answer the same relative-compass questions and measure their agreement with the automatically generated labels; if agreement is far from perfect, the benchmark itself is ambiguous and the model scores cannot be interpreted as a pure measure of compass reasoning.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-7B and LLaVA-13B model family that is evaluated and then fine-tuned."},{"cited_title":"Gpt-4o mini: A smaller, cheaper ai model,","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4o-mini, one of the stronger spatial models whose compass accuracy still collapses to near chance."},{"cited_title":"Ali iconfont,","cited_arxiv_id":null,"evidence_quote":"Source of the high-resolution icon images used to build the icon-type CDR questions."},{"cited_title":"Visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Visual instruction tuning method and the LLaVA model line that the fine-tuning experiments build on."}],"review_version":1}