{"id":"14b2ef03-dfe2-4a16-a2f7-a99d4c2c688b","arxiv_id":"2411.10857","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A generative vision-language model finetuned on remote-sensing data is claimed to beat baseline models on the RSVQAxBEN benchmark, but the supporting experimental details are missing.","lead":"The paper describes a method to teach a large vision-language model to answer questions about satellite images, using remote-sensing pretraining and prompt-based finetuning. It reports higher scores than three unnamed baselines on a standard remote-sensing question benchmark, but supplies almost no implementation details.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's 'state-of-the-art' claim is unverifiable because the three baselines are unnamed, uncited, and absent from the paper's own related-work survey, so the central superiority claim cannot be tested.","rationale":"The reader's weakest_assumption correctly identifies the undefined baselines and the unspecified pretraining dataset as critical gaps. I agree that the comparison premise is unsupported, but I would sharpen the concern: the paper's own related-work section names concrete remote-sensing LVLMs that should have been the baselines, and their absence from Table 1 makes the 'state-of-the-art' claim not merely unreproducible but internally inconsistent with the paper's survey. I also flag a distinct methodological ambiguity in Eqs. (3.4)/(3.6)/(3.8) regarding whether training is sequential or joint. None of these points is an ad hominem or a disagreement with the field's consensus; they are all internal consistency and evidence-quality issues. Because the central empirical claim cannot be checked without the missing model identity, baseline definitions, and code, I do not see a path to acceptance in the current form. The reader's REJECT verdict therefore stands; no verdict change is needed.","tokens_in":10043,"tokens_out":4407,"duration_ms":44743,"concrete_test":"Require the authors to identify the base LVLM and release code/configs, then re-run Table 1 on the same RSVQAxBEN splits with the models the paper itself cites as related state of the art—GeoChat, SkyEyeGPT, H2RSVLM, and RSGPT—using their official checkpoints or published numbers, plus a published discriminative RSVQA baseline. If the proposed method does not outperform these named systems, the 'state-of-the-art' claim fails. As a minimal analytical check, ask the authors to specify which loss is actually optimized: the sequential two-stage objective described in Section 3.3 or the joint objective in Eq. (8).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4.4, Table 1) is that the proposed generative LVLM with domain-adaptive pretraining and prompt-based finetuning outperforms state-of-the-art baselines. This claim rests on the premise that the comparators in Table 1 are legitimate, properly tuned state-of-the-art systems. The paper does not establish this: Section 4.1 names 'Baseline-1', 'Prompt-RSVQA', and 'Discriminative-RSVQA' but gives no model identity, no citations, no architecture, no training details, no hyperparameters, no error bars, and no code or data release. Meanwhile, Section 2.2 itself reviews GeoChat [21], SkyEyeGPT [22], H2RSVLM [26], RSGPT [28], and Ground Remote Alignment [27] as existing remote-sensing LVLMs, yet none appears in the comparison. A superiority claim against three unnamed stand-ins is not evidence of state-of-the-art performance. A second, related defect is that the base LVLM used by the proposed method is never identified (Section 3.2 says only 'a pre-trained LVLM'), so the reported 90.2/84.9/75.4 numbers cannot be reproduced or interpreted. There is also an internal inconsistency: Section 3.3 describes two sequential training stages, but Eq. (8) presents a joint objective L_total = λ1·L_pretrain + λ2·L_finetune; if the stages are sequential, the joint loss is not what is optimized, and if jointly optimized, the 'two-step' description is inaccurate. The experimental evidence for the paper's main claim is therefore unverifiable as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generative Large Vision-Language Model (LVLM) approach for Remote Sensing Visual Question Answering (RSVQA). The method consists of two training stages: domain-adaptive pretraining on remote sensing data and prompt-based finetuning, followed by beam-search decoding. The authors evaluate on RSVQAxBEN and report performance across yes/no, multiple-choice, and open-ended question types (Table 1), an ablation study of the two training components (Table 2), and a human evaluation of output quality (Table 3). The central claim, stated in Section 4.4, is that the proposed method outperforms three baselines on all question types and achieves the highest human ratings.","tokens_in":10331,"tokens_out":3936,"duration_ms":42868,"significance":"If the empirical results were verifiable, the paper would provide a useful data point on adapting generative LVLMs to the remote sensing domain, particularly the combination of domain-adaptive pretraining and prompt-based finetuning. The manuscript is clearly structured and evaluates on a public benchmark, and it reports both automatic metrics and human judgments. However, the central contribution cannot be assessed as written: the three baselines are not defined, the base LVLM is never identified, the training objective is internally inconsistent, and no experimental details (splits, hyperparameters, variance, statistical tests) are provided. These omissions are load-bearing, not cosmetic, because the paper's only substantive claim is empirical superiority. The significance of the work therefore remains unestablished.","major_comments":[{"comment":"The comparative claim is unverifiable because the three baselines (Baseline-1, Prompt-RSVQA, Discriminative-RSVQA) are introduced by name only, with no model identity, architecture, training procedure, citations, or hyperparameters. The manuscript's own related-work section (§2.2) reviews GeoChat [21], SkyEyeGPT [22], H2RSVLM [26], RSGPT [28], and Ground Remote Alignment [27], yet none of these appears in Table 1. Since the paper's central claim is 'superior performance compared to state-of-the-art baselines,' the comparison must be against identifiable, properly configured systems; as written, it is not.","section":"§4.1, Table 1"},{"comment":"The base LVLM is never identified: Section 3.2 only states that the model uses 'a pre-trained LVLM' with a visual encoder and a language decoder. No model name, parameter scale, architecture details, or pretraining corpus are given. Without this information, the reported numbers in Tables 1–3 cannot be reproduced, and the contribution of the proposed domain-adaptive pretraining and prompt-based finetuning cannot be isolated from the choice of the base model.","section":"§3.2, Eqs. (2)–(3)"},{"comment":"There is an internal inconsistency between the described training procedure and the stated objective. Section 3.3 describes a two-step sequential strategy (domain-adaptive pretraining followed by prompt-based finetuning), with separate losses in Eqs. (4) and (6). Section 3.5 then defines a joint objective L_total = λ1·L_pretrain + λ2·L_finetune. If the stages are sequential, no joint loss is ever optimized; if Eq. (8) is actually optimized, then the two-step description is inaccurate. The values of λ1, λ2, and beam size k are also never specified, so the method is underdetermined.","section":"§3.3, §3.5, Eq. (8)"},{"comment":"All reported results are point estimates with no error bars, confidence intervals, statistical significance tests, number of runs, random seeds, or dataset split details. The performance gaps in Table 1 are modest (e.g., 90.2 vs. 88.3 yes/no accuracy, 75.4 vs. 72.6 open-ended F1), so without variance information the claimed superiority may not be significant. Similarly, the ablation differences in Table 2 (e.g., 90.2 vs. 88.6) could fall within run-to-run noise. The manuscript does not report how the RSVQAxBEN data were split into training and test sets.","section":"Tables 1–3"},{"comment":"The domain-adaptive pretraining dataset D = {(I_i, Q_i, A_i)} is described only as 'curated,' with no source, size, composition, or explicit statement that it is disjoint from the RSVQAxBEN test split. Because the pretraining data include question-answer pairs, the possibility of overlap with the evaluation data is not addressed. The reported contribution of domain-adaptive pretraining (Table 2) is therefore uninterpretable without clarity on this dataset.","section":"§3.3, dataset D"},{"comment":"The human evaluation is insufficiently described: the number of annotators, their expertise, the annotation instructions, the sampling procedure for the 100 questions, and inter-annotator agreement are all omitted. All scores are reported as averages with no variance or statistical tests. Given the small score differences (e.g., 4.4 vs. 4.1 for correctness), the claim that the method produces 'more accurate, relevant, and fluent' answers is not supported as reported.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The text attributes a dataset named 'DDFA V' to reference [23], but reference [23] is the FAIR1M benchmark paper; the dataset name and citation do not match.","section":"§2.2"},{"comment":"References [6] and [7] are identical (both are Wornow et al., 'The shaky foundations of clinical foundation models'), so the related-work citations need to be corrected and de-duplicated.","section":"References"},{"comment":"Reference [16] has a garbled title, with multiple unrelated phrases concatenated; it should be replaced with the actual title of the cited work.","section":"References"},{"comment":"The prompt template 'Given the image features V, answer the question: Q' uses the formal feature-vector symbol V rather than a natural-language description; the authors should clarify whether V is replaced by a caption, a feature summary, or something else when the prompt is presented to the model.","section":"Eq. (5)"},{"comment":"The abstract and introduction emphasize that the generative approach avoids 'predefined answer categories,' yet the evaluation includes multiple-choice accuracy, which by definition uses predefined options; this point should be clarified to avoid appearing contradictory.","section":"Introduction and Abstract"}],"recommendation":"reject","confidential_remarks":"The manuscript is well below the bar for publication in its current form. The empirical core is unverifiable because the baselines and the base model are unspecified, and the training description is internally inconsistent. These are not local issues; they affect the central claim of state-of-the-art performance. In addition, the reference list contains duplicates, a mismatched dataset citation, and several entries that are never discussed in the text, which makes the provenance of the related-work claims difficult to check. I would not encourage resubmission without a fundamentally reworked experimental section that identifies all models, provides full implementation details, reports variance and significance, and compares against the existing remote-sensing LVLMs reviewed in Section 2.2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is easy to size up: the method is the standard LVLM recipe—domain-adaptive pretraining plus prompt-based finetuning on a remote sensing VQA dataset—and the paper's own related work section shows that GeoChat, SkyEyeGPT, H2RSVLM, and RSGPT already do this kind of thing. So the contribution is incremental at best. What the paper does well is write clearly and survey the right literature; if you need a quick map of the RS-LVLM space, Section 2 is fine.\n\nThe soft spots are load-bearing. Table 1 lists three baselines—Baseline-1, Prompt-RSVQA, Discriminative-RSVQA—but never says which models they are, where the numbers came from, or even cites them. The paper's own related work reviews the state-of-the-art generative RS LVLMs, yet none of those appears in the comparison. That makes the 'superior performance' claim unverifiable, and it is the paper's main claim. The base LVLM is never identified (Section 3.2 just says 'a pre-trained LVLM'), so the 90.2/84.9/75.4 numbers cannot be reproduced or interpreted. There is also a real internal inconsistency: Section 3.3 describes two sequential training stages, but Eq. (8) gives a joint objective combining both losses; if the stages are sequential, that joint loss is not what is optimized, and if they are joint, the 'two-step' description is wrong. No error bars, no hyperparameters, no dataset splits, no code, no data release. The ablation removes components from the same pipeline without independent validation, and the human evaluation is a 100-question subset with no inter-annotator agreement and no error bars.\n\nIn fairness, the paper is not nonsense—it is just a very incomplete report of a standard experiment. The direction (generative LVLMs for open-ended RSVQA) is reasonable, and the authors have read the right papers. But the central claim is not checkable, and there is no new dataset, mechanism, or analysis to compensate.\n\nI would not bring this to reading group, and I would not cite it. If this crossed my desk, I would desk reject it rather than send it to referees: the missing details are so extensive that a review would just become a list of 'please specify everything.' The authors should be told what is missing and invited to resubmit with real baselines, an identified backbone, and released code and data.","headline":"A standard LVLM-for-RSVQA recipe whose only empirical claim is unverifiable because the baselines, backbone, and training details are all unspecified.","tokens_in":10927,"tokens_out":3929,"would_cite":false,"duration_ms":37075,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative LVLM with domain-adaptive pretraining and prompt-based finetuning outperforms prior RSVQA models on every question type in the RSVQAxBEN benchmark.","keywords":["Large vision-language models","Remote sensing visual question answering","Domain-adaptive pretraining","Prompt-based finetuning","RSVQAxBEN","Generative VQA","Satellite imagery","Open-ended question answering"],"falsifier":"Re-run the exact RSVQAxBEN split with publicly available, properly tuned baselines (for example the released checkpoints of GeoChat or SkyEyeGPT) over multiple seeds and report confidence intervals; if any baseline matches or beats 90.2% yes/no, 84.9% multiple-choice, and 75.4% open-ended F1, the paper's superiority claim falls. A second check is to audit the pretraining dataset $\\mathcal{D}$ for overlap with the test split; any leakage would invalidate the numbers.","tokens_in":9787,"feed_emoji":"🛰️","tokens_out":6038,"duration_ms":54294,"temperature":0.7,"pith_summary":"This paper tries to establish that a generative large vision-language model, adapted to satellite imagery in two steps, is the right tool for remote sensing visual question answering. The two steps are domain-adaptive pretraining on remote sensing data and prompt-based finetuning for the question-answering task, which together let the model produce free-form natural language answers instead of choosing among predefined categories. On the RSVQAxBEN benchmark, the method reports 90.2% yes/no accuracy, 84.9% multiple-choice accuracy, and 75.4% open-ended F1, beating the three baselines it compares against on all three question types. A human evaluation on 100 questions also rates its answers highest on correctness, relevance, and language quality. If the claims hold, generative LVLMs offer a single streamlined pathway for open-ended remote sensing question answering without separate task-specific heads.","feed_headline":"Generative LVLM tops all question types in satellite-image VQA","feed_subtitle":"Domain-adaptive pretraining plus prompt finetuning beats separate-encoder and discriminative baselines on RSVQAxBEN.","key_machinery":"The load-bearing mechanism is a two-step training recipe applied to a pretrained LVLM composed of a visual encoder $f_{\\text{enc}}$ and a language decoder $g_{\\text{dec}}$. First, domain-adaptive pretraining minimizes the negative log-likelihood of answer tokens on a curated remote sensing dataset $\\mathcal{D}$; second, prompt-based finetuning refines the model on prompts of the form \"Given the image features V, answer the question: Q\". Inference uses beam search over the token-level conditional probabilities, and the total objective is $\\lambda_1 \\mathcal{L}_{\\text{pretrain}} + \\lambda_2 \\mathcal{L}_{\\text{finetune}}$. The role of this machinery is to inject remote sensing-specific visual knowledge into a general-purpose LVLM and then steer it toward concise answer generation, which is what the paper argues produces the reported gains over prompt-only and discriminative baselines.","core_discovery":"The central claim is that a generative LVLM trained with domain-adaptive pretraining followed by prompt-based finetuning achieves state-of-the-art performance on RSVQAxBEN across yes/no, multiple-choice, and open-ended questions. The paper frames the task as maximizing the conditional probability $p(A|I,Q;\\theta)$ of a natural language answer $A$ given satellite image $I$ and question $Q$, generated token by token by a language decoder conditioned on visual features from an encoder. The implementation reported in Table 1 reaches 90.2% yes/no accuracy, 84.9% multiple-choice accuracy, and 75.4% open-ended F1, and the ablation in Table 2 shows that removing either training step lowers all three numbers. The authors conclude that both domain adaptation and prompt engineering contribute, and that the generative formulation handles open-ended questions more flexibly than discriminative models with rigid answer classes.","pith_inferences":["If the reported gaps replicate, the main source of gain is probably the domain-adaptive pretraining step rather than the generative architecture itself; a direct test would be to freeze the base LVLM and vary only the pretraining corpus.","The paper leaves the pretraining dataset $\\mathcal{D}$ unspecified; a natural extension is to measure how performance scales with the size and diversity of that corpus, and whether mixing in non-remote-sensing data helps or hurts.","Because the baselines are not defined or cited, an independent replication should first pin down what \"state-of-the-art\" means on RSVQAxBEN; until then, the practical implication is that generative LVLMs are competitive, not that this specific configuration is uniquely optimal.","The same two-step prompting recipe could be tested on temporal or multispectral inputs, where the visual encoder would need extension; the paper's formulation already allows $C$ spectral bands in $I\\in\\mathbb{R}^{H\\times W\\times C}$."],"forward_implications":["Open-ended RSVQA no longer needs a fixed answer vocabulary; any question whose answer can be expressed in language is in principle addressable by the same generative model.","The two-step recipe should transfer to other remote sensing vision-language tasks such as captioning, land-use classification by description, and region-grounded dialogue, since it only changes the prompt and finetuning data.","Domain-adaptive pretraining plus prompt finetuning may reduce the amount of labeled VQA data needed for a new geographic area or sensor type, because the pretraining step carries the domain knowledge.","Benchmark comparisons in RSVQA should include generative LVLM baselines alongside discriminative classifiers, since the best reported results now come from the generative family."],"supporting_citations":[{"why":"Supplies the RSVQAxBEN benchmark, including the image-question-answer triples and the train/test split on which all reported accuracy and F1 numbers are computed.","marker":"[3]"},{"why":"Defines the RSVQA task for remote sensing data and establishes the yes/no, multiple-choice, and open-ended question evaluation scheme that this paper inherits.","marker":"[2]"},{"why":"GeoChat is cited as a prior grounded remote sensing LVLM, motivating the use of vision-language models for conversational satellite image analysis.","marker":"[21]"},{"why":"SkyEyeGPT is cited as a prior instruction-tuned remote sensing LVLM, motivating the domain-adaptive pretraining and instruction-style finetuning used in the method.","marker":"[22]"}],"fun_headline_variants":["Generative LVLM achieves SOTA on satellite VQA","Two-step training lifts LVLM to top on RSVQA","LVLM answers open-ended satellite questions better","Satellite VQA: generative model beats all baselines","Domain-adaptive pretraining boosts LVLM VQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison's validity rests on the unstated assumption that the three baselines in Section 4.1 are legitimate, well-tuned state-of-the-art systems, since none of them is defined, cited, or given error bars; the paper also assumes that the curated pretraining dataset $\\mathcal{D}$ exists, is large enough to drive adaptation, and is disjoint from the test split.","fun_headline_variants_meta":{"raw":{"variants":["Generative LVLM achieves SOTA on satellite VQA","Two-step training lifts LVLM to top on RSVQA","LVLM answers open-ended satellite questions better","Satellite VQA: generative model beats all baselines","Domain-adaptive pretraining boosts LVLM VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1681,"prompt_tokens":908,"completion_tokens":773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":695}},"tokens_in":524,"tokens_out":773,"duration_ms":8784,"temperature":1.0,"reasoning_tokens":695,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:12:21.036946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact RSVQAxBEN split with publicly available, properly tuned baselines (for example the released checkpoints of GeoChat or SkyEyeGPT) over multiple seeds and report confidence intervals; if any baseline matches or beats 90.2% yes/no, 84.9% multiple-choice, and 75.4% open-ended F1, the paper's superiority claim falls. A second check is to audit the pretraining dataset $\\mathcal{D}$ for overlap with the test split; any leakage would invalidate the numbers.","supporting_citations":[],"review_version":1}