Pith. sign in

REVIEW 4 major objections 4 minor 37 references

When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a model's agreement across re-renderings certifies correctness only above a threshold set by error diffuseness, and that training on a model's own consensus raises agreement while lowering accuracy.

desk verdict Serious, honest paper with a novel measurement instrument, but the headline 're-rendering beats resampling' is confounded by decoding temperature, and the Limitations contradict the appendix on the diffuseness threshold. read the letter →

arxiv 2608.05670 v1 pith:P26EKDN4 submitted 2026-08-06 cs.LG

Rasul Khanbayov , Hasan Kurban This is my paper · ORCID
classification cs.LG
keywords vision-languagemodelsself-consistencychartquestionansweringcalibrationtest-timeaggregationrender-equivalencesetsagreement-accuracycouplingconsensusself-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to settle a question that much of the field has assumed away: when does a model's agreement across perturbed inputs actually track whether its answer is right? It builds a measurement instrument, RENDEQ, whose render-equivalence sets redraw the same chart data under cosmetic styles that leave the underlying values unchanged, so every image shares a programmatically exact answer and meaning preservation holds by construction rather than by assumption. Measuring the agreement-accuracy coupling on three open-weight vision-language models (Qwen2.5-VL-7B, Qwen2.5-VL-3B, InternVL2-8B) across three independent instantiations, the paper reports that re-rendering beats resampling on accuracy and reliability, that agreement beats an evidence-carrying baseline on two of three models, and that one style factor, the plotting library, dominates the dispersion. Its most consequential finding is that fine-tuning on a model's own cross-render consensus lowers accuracy in all five replication runs, the opposite sign of a published natural-image result. The paper's account of why is its central claim: agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.

What carries the argument

The central object is the render-equivalence set: a family of $K$ images obtained by re-rendering one dataset under style configurations (plotting library, palette, theme, fonts, gridlines, resolution) that leave the encoded data unchanged, so every image shares a programmatically exact answer and any answer change is attributable to the model rather than to the perturbation. From each set the paper defines two label-free quantities: render-equivalence agreement (REA), the modal share of answers, which estimates the concentration $\pi$, and render marginalization (RM), the modal or median answer, which aggregates it. The load-bearing identity is the certification condition of Proposition 5(iii): under exchangeable renderings, when errors are diffuse at level $L$, the leading wrong answer carries at most $(1 - P(a^\star))/L$ of the mass, so a confidently wrong consensus above agreement $1/L$ is impossible and agreement above $1/L$ certifies the modal answer as correct; when all error mass sits on one wrong answer ($L = 1$) the threshold is $1$ and nothing short of unanimity certifies. Two supporting results carry the empirics: Proposition 4(ii) shows modal REA is upward-biased for concentration at finite $K$, which motivates the unbiased pairwise alternative REApair, and Proposition 7 bounds total cross-render disagreement by the sum of per-factor disagreement rates, which licenses reading the per-factor lifts of Section 6.3 as a causal attribution of which style factor carries the dispersion.

What would settle it

Run the library-held-fixed ablation the paper identifies as missing: generate render-equivalence sets whose $K$ renderings all use a single plotting backend (matplotlib only, then plotly only) and rerun the headline comparisons. If RM's accuracy margin over self-consistency and REA's AUROC margin over mean token log-probability vanish or change sign, the measured coupling is an artifact of backend sensitivity rather than a property of render equivalence; if the margins persist, the dominant-factor attribution is confirmed as real but secondary.

Watch

Extended reading notes

Core claim

The paper's central claim is that render-equivalence sets make the agreement-accuracy coupling exactly measurable, and that once measured the coupling turns out to be real, useful, and unsafe to optimize. A scientific figure is the output of a rendering program applied to data, so redrawing it under styles that leave the data unchanged yields images that are semantically equivalent by construction and share a programmatic answer key for the figure and for every intermediate read; agreement across these renderings is the model's own instability and nothing else. The analysis shows that modal agreement (REA) estimates the concentration $\pi$ of the model's style-induced answer distribution, not its correctness, and is upward-biased at finite $K$, while the modal answer across renderings (RM) aggregates the distribution and can be wrong whenever the model has a systematic misread. The key theoretical result is a certification condition: if a model's errors are diffuse at level $L$, with the leading wrong answer carrying at most $(1 - P(a^\star))/L$ of the mass, then agreement above $1/L$ certifies that the modal answer is correct, and at $L = 1$ — a single systematic misread — nothing certifies. The paper reports four empirical results consistent with this account: render marginalization beats self-consistency and single-render decoding on every model and every replicated instantiation; cross-render agreement beats mean token log-probability on two of three models and ties on the third; the plotting library accounts for more than double the next factor's dispersion; and fine-tuning on a model's own cross-render consensus lowers accuracy in all five replicate runs, the opposite of a published natural-image result that the certificate-destroying mechanism explains.

Load-bearing premise

The central analysis assumes exchangeable renderings — that a model's answers across renderings are i.i.d. draws from one style-induced distribution with a unique mode — which the paper admits real style spaces violate, since they are finite and correlated and its own data show the plotting library dominating dispersion; the separate perception-versus-reasoning split also rests on an intermediate-read query whose reliability the authors state was never validated.

Editorial extensions

If this is right

  • Render marginalization becomes a test-time compute strategy for chart question answering: aggregating answers across $K$ re-renderings beats both single-render decoding and fixed-image self-consistency at equal call count, on every model and every replicated instantiation tested.
  • Agreement is a competitive label-free reliability signal: cross-render agreement beats mean token log-probability on two of three models, ties on the third, and a cross-validated logistic combination of the two signals raises AUROC on every model.
  • Practitioners with a perturbation budget should vary the plotting backend first: the library factor's lift is more than twice the next-largest factor's and roughly an order of magnitude above the noise floor.
  • Consensus self-training is regime-dependent: fine-tuning on a model's own cross-render majority raises hold-out agreement while lowering hold-out accuracy in all five runs, so the published natural-image gains do not transfer to re-rendered figures.
  • The certification threshold is portable in principle: diffuseness $L$ can be estimated offline on any generator with programmatic ground truth and the threshold $1/L$ applied label-free elsewhere, and in practice it is demanding — median $\hat{L}$ on wrong-mode instances here is 1.33 to 1.50.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The boundary condition likely generalizes beyond charts: any training objective that rewards agreement directly — consistency regularization, entropy minimization, self-consistency fine-tuning — should degrade accuracy whenever a model's errors are systematic rather than idiosyncratic, in any domain where the perturbation family preserves meaning exactly.
  • Because the plotting library dominates the dispersion, an untested corollary is that within-library re-rendering (the library-held-fixed ablation the paper leaves missing) could substantially shrink REA's and RM's advantages; if it did, the measured coupling would be closer to backend sensitivity than to a general property of render equivalence.
  • The paper argues, but does not measure, that exact semantic equivalence is what buys the advantage over ordinary pixel augmentation; adding the missing approximate-perturbation arm would turn that argued difference into a quantified one.
  • The certification threshold $1/L$ assumes diffuseness transfers from the generator to deployment data, which the paper does not test; measuring $\hat{L}$ on the out-of-style split or on real charts would show whether agreement above $1/\hat{L}$ actually certifies correctness there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces RENDEQ, a generator of render-equivalence sets for chart figures, in which the same data are redrawn under different cosmetic styles so that the resulting images are semantically equivalent by construction and share a programmatically exact answer. It defines render-equivalence agreement (REA) and render marginalization (RM), derives formal results about the relationship between agreement and correctness under an exchangeability assumption, and reports experiments on three open-weight VLMs across three independently generated instantiations. The headline empirical claims are that RM beats single-render decoding and self-consistency on accuracy, REA beats self-consistency on reliability and beats mean token log-probability on two of three models, that the plotting library dominates the style-induced dispersion, and that fine-tuning on a model's own consensus reduces accuracy. The paper is unusually transparent: it discloses a rendering-pipeline bug that invalidated an intermediate version, reports corrected numbers, and explicitly lists unresolved limitations.

Significance. If the central empirical claim holds, the paper makes a valuable contribution: a measurement instrument that makes agreement and correctness simultaneously observable under exactly meaning-preserving perturbation, with programmatic ground truth and individually togglable style factors. The paper's empirical discipline is a genuine strength: three independent generator instantiations, cluster-bootstrap confidence intervals, a publicly disclosed and fixed rendering bug, recomputed calibration and K-ablation tables on the corrected pool, and code/data release. The theoretical results are clearly derived from stated assumptions and are not fitted to the outcomes. However, the significance is conditional on resolving a load-bearing confound in the headline render-versus-resample comparison and on reconciling several self-reported contradictions between the main text and the Limitations.

major comments (4)
  1. [Sec. 6.1, 6.2, Tables 3-4, Fig. 2a, Sec. 7] The central claim that re-rendering beats resampling is not identified, because the render arms use greedy decoding (temperature 0) while the resampling arms use temperature 0.7. The manuscript itself states the caveat in Sec. 6.2: "the main RM runs decode at temperature 0 while SC needs 0.7 to vary at all." Decoding temperature changes both accuracy and answer diversity, so the reported +4.8 to +6.3 accuracy-point advantage of RM over SC and the +0.061 to +0.157 AUROC advantage of REA over B3 do not isolate perturbation type from sampling temperature. B3 is described in Sec. 6.1 as fixed-image self-consistency over K=8 samples, the same procedure as SC in Table 4, yet its temperature is never stated. This is load-bearing because the abstract's first empirical claim and Sec. 7's "Render marginalization beats every control we matched it against" rest on exactly these comparisons. Please run RM and REA at temperature 0.7 (and, if feasible, SC under a configuration that matches the render arms) and report both configurations; without this, the render-versus-resample claim is not established.
  2. [Sec. 4.2, Sec. E, Limitations] Theorem 5(iii) defines the certifying threshold 1/L, and Sec. 4.2 reports that L was estimated on every wrong-mode instance of the replicated pool (median 1.33-1.50), with a full distribution in Sec. E. The Limitations, however, state that "Theorem 5(iii) is stated in terms of a diffuseness level L that we have not yet measured." These statements directly contradict each other. Moreover, the L estimates in Sec. E are computed on the same generator used for the rest of the experiments, and the threshold is never applied to a held-out set with a measured false-certification rate, so the boundary-condition claim in the abstract is currently an untested prediction rather than a validated certificate. Please resolve the contradiction and either provide a held-out threshold evaluation or clearly label the certificate as a stylized prediction.
  3. [Sec. 3, Sec. E.1, Limitations] The paper claims exact ground truth for "every intermediate read" and uses the intermediate-read query to partition errors into perception and reasoning failures, concluding that roughly four-fifths of instances have at least one bad intermediate read and that true reasoning failures are rare. The Limitations, however, state that the perception-versus-reasoning split rests entirely on an intermediate-read query whose own reliability was never validated. A wrong answer to this query could reflect output-format errors, parsing failures, or a different internal read rather than the model's actual perception error, so the decomposition is not established as measurement. Please either validate the read-query construct (for example against a programmatic check of the read values and their alignment with the final answer) or remove the perception/reasoning claims from the paper's contributions and abstract.
  4. [Sec. 4, Assumption 2, Sec. 6.3] The theoretical results (Prop. 4, Prop. 5, Prop. 6, Prop. 7, and the threshold 1/L) assume i.i.d. renderings conditional on (D,q), but the paper admits that real style spaces are finite and correlated, and Sec. 6.3 shows that one factor, the plotting library, dominates the dispersion. The empirical definitions of REA and RM do not require the i.i.d. assumption to be well-defined, but the upward-bias result, the certification threshold, and the factor-decomposition bound are derived under it. Please add a robustness analysis that does not rely on exchangeability, for example by blocking on the plotting library or by obtaining cluster-bootstrap confidence intervals over style configurations, and state explicitly which conclusions in Sec. 7 would change if Assumption 2 is replaced by the actual correlated style process.
minor comments (4)
  1. [Sec. 6.1 and Table 4] B3 and SC appear to be the same fixed-image self-consistency baseline, but B3's decoding temperature is not stated in Sec. 6.1 even though Table 4 reports SC at temperature 0.7. Please state B3's temperature explicitly and align the two descriptions.
  2. [Fig. 2a] The caption says filled markers denote accuracy and hollow markers denote reliability, but the marker glyphs are not identified in the figure itself. Please add a legend or an explicit symbol description so the reader can decode the plot without the caption alone.
  3. [Table 1 and Sec. 6.3] The factor names in Table 1 (ticks, aspect ratio, legend) differ from the names in the lift table (tick_density, bar_orient, pie_start_angle). Please harmonize the naming so the style factors are traceable from the axis definition to the attribution analysis.
  4. [Sec. 6.1 and Limitations] The paper states that semantic entropy, P(True), and an approximate-perturbation arm were not run. Since these are standard label-free signals and the absence of an approximate-perturbation arm is explicitly acknowledged as leaving the exact-equivalence delta unmeasured, please add a sentence in Sec. 6.1 explaining whether the omissions are due to cost, scope, or methodology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical results follow from explicit assumptions, and the empirical claims are measured against programmatic ground truth rather than reduced to their own inputs.

full rationale

The derivation chain is self-contained. Proposition 4, Theorem 5, Corollary 6, and Proposition 7 are proved from Assumptions 2 and 3; REA is a modal-frequency statistic and RM is a mode/median aggregation, so the correctness target (whether the aggregated mode equals the programmatic answer a*) is not defined in terms of REA. Theorem 5(iii) is an analytic implication of the definition of diffuseness L, not a fitted prediction, and the paper later discloses that Theorem 5(iii) is stated for an L that has not yet been measured (Limitations), which is an inconsistency with the L-hat estimate in Sec. 4.2/E, not a circular reduction. Sec. G explicitly flags the earlier K=24 majority-target computation as circular and corrects it by holding the target fixed, so the current K-ablation does not repeat that error. The RM-versus-SC temperature mismatch in Sec. 6.2 is a methodological confound (greedy rendering vs. temperature-0.7 resampling), but the render and resampling arms are not defined in terms of each other's outputs. The L-based threshold and family-level diffuseness discussion are post-hoc interpretations of measured data, not parameters fitted to force a stated prediction. No load-bearing self-citation or imported uniqueness claim appears; external benchmarks and baselines are used only as comparators. The admitted limitations (untested out-of-style split, unvalidated intermediate-read query, missing approximate-perturbation arm) concern external validity and completeness, not circularity of the derivations.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central measurement depends on the i.i.d. rendering assumption, the factored style-space assumption, and the unvalidated intermediate-read query; K, tau, and the estimated diffuseness L are hand-chosen or data-derived values that shape the reported magnitudes.

free parameters (4)
  • K (renderings per instance) = 8
    Default budget across all experiments; K-ablation (Sec. G) shows partial saturation rather than full convergence; chosen by hand, not fitted.
  • Numeric tolerance tau = 5%
    Relative tolerance for numeric answers; authors call it load-bearing and test sensitivity in Sec. G (1%, 2%, 5%, 10%).
  • Diffuseness level L (estimated) = median 1.33-1.50 across models
    Data-derived estimate from wrong-mode instances (Sec. E) used to set the certifying threshold 1/L; measured, not fitted, but computed on the same generator.
  • LoRA rank and epochs = r=8, 3 epochs
    Fine-tuning configuration for the consensus self-training experiment; chosen by hand and unvaried.
assumptions (6)
  • domain assumption Assumption 2: Conditional on (D,q), answers are i.i.d. draws from P induced by the style distribution and the model; unique mode where referenced.
    Invoked in Sec. 4 for Prop. 4, 5, 6 and Lemma 9, 10; authors call it an idealization and acknowledge it fails when style-invariant bias exists.
  • domain assumption Assumption 3: The style space is a product over independent nuisance factors.
    Used in Sec. 4.3 Prop. 7 for the factor decomposition; RENDEQ enforces it by construction, but real style spaces are correlated.
  • domain assumption The renderer R is deterministic and style factors are semantically irrelevant, so all images in a render-equivalence set share the exact answer a*(D,q).
    Foundation of render-equivalence sets (Sec. 3); if a plotting library change alters the encoded data perceptibly, equivalence is not by construction.
  • ad hoc to paper Intermediate-read queries provide accurate ground truth for the perception/reasoning error decomposition.
    The error decomposition in Sec. E.1 rests on a separate query whose reliability the authors state was never validated (Limitations).
  • standard math Regularity conditions for median aggregation: unique median and positive continuous density at the median.
    Assumed for Lemma 10 sample-median CLT in Sec. B.
  • domain assumption The correctness target is programmatically exact and shared across all renderings.
    Defines a* in Sec. 3; without exact shared ground truth, the measurement instrument collapses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering." pith.science (2026). https://pith.science/paper/P26EKDN4

@misc{pith2026260805670,
  author       = {Pith},
  title        = {Pith review of: When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P26EKDN4}},
  note         = {Machine review of arXiv:2608.05670}
}
read the original abstract

A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 15 canonical work pages

  1. [1]

    Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation.arXiv preprint arXiv:2411.11919, 2024

    Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation.arXiv preprint arXiv:2411.11919, 2024

    arXiv 2024

  2. [2]

    Questioning the stability of visual question answering

    Amir Rosenfeld, Neta Glazer, and Ethan Fetaya. Questioning the stability of visual question answering. arXiv preprint arXiv:2511.11206, 2025

    arXiv 2025

  3. [3]

    Efficient test-time scaling for small vision-language models

    Mehmet Onurcan Kaya, Desmond Elliott, and Dim Papadopoulos. Efficient test-time scaling for small vision-language models. InInternational Conference on Learning Representations, volume 2026, pages 146270–146312, 2026

    work page 2026

  4. [4]

    Multi-llm debate: Framework, principals, and interventions.Advances in Neural Information Processing Systems, 37:28938–28964, 2024

    Andrew Estornell and Yang Liu. Multi-llm debate: Framework, principals, and interventions.Advances in Neural Information Processing Systems, 37:28938–28964, 2024

    2024

  5. [5]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    arXiv 2022

  6. [6]

    Consistency and uncertainty: Identifying unreliable responses from black- box vision-language models for selective visual question answering

    Zaid Khan and Yun Fu. Consistency and uncertainty: Identifying unreliable responses from black- box vision-language models for selective visual question answering. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10854–10863. IEEE, 2024

    2024

  7. [7]

    Decompose and compare consistency: Measuring vlms’ answer reliability via task-decomposition consistency comparison

    Qian Yang, Weixiang Yan, and Aishwarya Agrawal. Decompose and compare consistency: Measuring vlms’ answer reliability via task-decomposition consistency comparison. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3613–3627, 2024

    work page 2024

  8. [8]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.arXiv preprint arXiv:2302.09664, 2023

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.arXiv preprint arXiv:2302.09664, 2023

    arXiv 2023

Show all 37 references
  1. [9]

    Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, 2024

    2024

  2. [10]

    Zoom consistency: A free confidence signal in multi-step visual grounding pipelines.arXiv preprint arXiv:2604.15376, 2026

    Keon Kim and Krish Chelikavada. Zoom consistency: A free confidence signal in multi-step visual grounding pipelines.arXiv preprint arXiv:2604.15376, 2026

    2026 arXiv

  3. [11]

    Vauq: Vision-aware uncertainty quantification for lvlm self-evaluation

    Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du, and Sharon Li. Vauq: Vision-aware uncertainty quantification for lvlm self-evaluation. InFindings of the Association for Computational Linguistics: ACL 2026, pages 26534–26550, 2026

    2026

  4. [12]

    Vl-calibration: Decoupled confidence calibration for large vision-language models reasoning

    Wenyi Xiao, XU Xinchi, and Leilei Gan. Vl-calibration: Decoupled confidence calibration for large vision-language models reasoning. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 44791–44815, 2026

    2026

  5. [13]

    Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23783–23793. IEEE, 2024

    2024

  6. [14]

    Figureqa: An annotated figure dataset for visual reasoning.arXiv preprint arXiv:1710.07300, 2017

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning.arXiv preprint arXiv:1710.07300, 2017

    2017 arXiv

  7. [15]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1516–1525. IEEE, 2020. 14 WHEN DOES CONSENSUS MEAN CORRECTNESS?Kurban Intelligence Lab

    2020

  8. [16]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Jia Qing Tan, Shafiq Joty, Enamul Hoque, et al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  9. [17]

    Chartverse: Scaling chart reasoning via reliable programmatic synthesis from scratch

    Zheng Liu, Honglin Lin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, Zhuoshi Pan, et al. Chartverse: Scaling chart reasoning via reliable programmatic synthesis from scratch. InProceedings of the 64th Annual Meeting of the Association for C...

    2026

  10. [18]

    Chart-rl: Generalized chart comprehension via reinforcement learning with verifiable rewards

    Xin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao, Zheng Wang, Yuying Wang, Dan Roth, and Chenyang Li. Chart-rl: Generalized chart comprehension via reinforcement learning with verifiable rewards. InProceedings of the First Workshop on Structured Understanding, Retrieval, an...

    2026

  11. [19]

    Chart-rvr: Reinforce- ment learning with verifiable rewards for explainable chart reasoning.arXiv preprint arXiv:2510.10973, 2025

    Sanchit Sinha, Oana Frunza, Kashif Rasul, Yuriy Nevmyvaka, and Aidong Zhang. Chart-rvr: Reinforce- ment learning with verifiable rewards for explainable chart reasoning.arXiv preprint arXiv:2510.10973, 2025

    2025

  12. [20]

    Chartgen: Scaling chart understanding via code-guided synthetic chart generation.arXiv preprint arXiv:2507.19492, 2025

    Jovana Kondic, Pengyuan Li, Dhiraj Joshi, Zexue He, Shafiq Abedin, Jennifer Sun, Ben Wiesel, Eli Schwartz, Ahmed Nassar, Bo Wu, et al. Chartgen: Scaling chart understanding via code-guided synthetic chart generation.arXiv preprint arXiv:2507.19492, 2025

    2025 arXiv

  13. [21]

    Unraveling the truth: Do vlms really understand charts? a deep dive into consistency and robustness

    Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, and Dan Roth. Unraveling the truth: Do vlms really understand charts? a deep dive into consistency and robustness. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1669...

    2024

  14. [22]

    Losing the plot: How vlm responses degrade on imperfect charts.arXiv preprint arXiv:2509.18425, 2025

    Philip Wootaek Shin, Jack Sampson, Vijaykrishnan Narayanan, Andres Marquez, and Mahantesh Halappanavar. Losing the plot: How vlm responses degrade on imperfect charts.arXiv preprint arXiv:2509.18425, 2025

    2025

  15. [23]

    On robustness and chain-of-thought consistency of rl-finetuned vlms.arXiv preprint arXiv:2602.12506, 2026

    Rosie Zhao, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang, Joerg Liebelt, and Arnab Mondal. On robustness and chain-of-thought consistency of rl-finetuned vlms.arXiv preprint arXiv:2602.12506, 2026

    2026 arXiv

  16. [24]

    Perception-r1: Advancing multimodal reasoning capabilities of mllms via visual perception reward

    Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Perception-r1: Advancing multimodal reasoning capabilities of mllms via visual perception reward. InInternational Conference on Learning Representations, volume 2026, pages 26868–26898, 2026

    2026

  17. [25]

    V-fat: Benchmarking visual fidelity against text-bias.arXiv preprint arXiv:2601.04897, 2026

    Ziteng Wang, Yujie He, Guanliang Li, Siqi Yang, Jiaqi Xiong, and Songxiang Liu. V-fat: Benchmarking visual fidelity against text-bias.arXiv preprint arXiv:2601.04897, 2026

    2026

  18. [26]

    Cdh-bench: A commonsense- driven hallucination benchmark for evaluating visual fidelity in vision-language models.arXiv preprint arXiv:2603.27982, 2026

    Kesheng Chen, Yamin Hu, Qi Zhou, Zhenqian Zhu, and Wenjian Luo. Cdh-bench: A commonsense- driven hallucination benchmark for evaluating visual fidelity in vision-language models.arXiv preprint arXiv:2603.27982, 2026

    2026

  19. [27]

    Do vision-language models see or guess? measuring and reducing textual-prior reliance with a phrasing-controlled benchmark.arXiv preprint arXiv:2606.10400, 2026

    Pratham Singla, Shivank Garg, Vihan Singh, and Paras Chopra. Do vision-language models see or guess? measuring and reducing textual-prior reliance with a phrasing-controlled benchmark.arXiv preprint arXiv:2606.10400, 2026

    2026 arXiv

  20. [28]

    On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5), 2010

    Ran El-Yaniv et al. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5), 2010

    2010

  21. [29]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InInternational conference on machine learning, pages 1321–1330. PMLR, 2017

    2017

  22. [30]

    selective prediction

    Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khy- athi Chandu. Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. InFindings of the Association for Computational Linguistics: ACL 2024...

    2024

  23. [31]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  24. [32]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  25. [33]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  26. [34]

    Equivalence guaranteed

    Gongyao Jiang and Qiong Luo. Chart-coca: Self-improving chart understanding of vision lms via code- driven synthesis and candidate-conditioned answering. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 1168–1178, 2025. APPE...

    2025

  27. [35]

    When the answer space has two elements and 𝐾 is odd the first inequality is an equality, and the middle quantity is strictly decreasing along odd𝐾

    Under any tie-breaking rule, Pr[̂ 𝑦RM≠𝑎 ⋆]≤Pr[Bin(𝐾,𝑝)≥ 𝐾 2] ≤𝑒 −2𝐾( 1 2−𝑝)2 . When the answer space has two elements and 𝐾 is odd the first inequality is an equality, and the middle quantity is strictly decreasing along odd𝐾. Proof. Let𝑊=|{𝑖∶̂ 𝑦 𝑖 ≠𝑎 ⋆}|∼ Bin(𝐾,𝑝) . If 𝑊 < 𝐾/...

  28. [36]

    indicators, each with expectation Pr[̂ 𝑦𝑖 =̂ 𝑦𝑗] = ∑𝑎𝑃(𝑎)2 =𝐴 for 𝑖≠𝑗 by independence and identical distribution, so 𝔼[REApair] =𝐴 regardless of 𝐾. For the sandwich,𝐴= ∑𝑎𝑃(𝑎)2≥𝑃(𝑚 ⋆)2 =𝜋 2 since𝜋2 is one of the summands, and𝐴= ∑𝑎𝑃(𝑎)𝑃(𝑎)≤ 𝜋 ∑𝑎𝑃(𝑎)=𝜋.□ 0 1 3 1 2 1 𝑍=0 𝑍=1 certi...

  29. [37]

    too many arguments

    Taking any𝑃 with𝜋 near1 and𝑚⋆≠𝑎 ⋆ on a set of instances of probability approaching one gives the final claim.□ C.4 Proof of Theorem 7 Let𝜃 and𝜃′ be independent draws with independent coordinates (Theorem 3). Define the hybrids 𝐻0 =𝜃 and𝐻𝑗 = (𝜃′ 1,…,𝜃 ′ 𝑗,𝜃𝑗+1,…,𝜃 𝐽), so 𝐻𝐽 =𝜃 ...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.