{"id":"f8c5c100-2e3a-4a48-a7e5-b974a1b3cb03","arxiv_id":"2608.11047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A deterministic pipeline generates financial tabular QA benchmarks with answers correct by construction and controllable difficulty.","lead":"V-FiLLM, a new benchmark generator, creates financial reasoning questions and verified answers from synthetic spreadsheets by sampling arithmetic computation trees. A team at ETH Zurich uses it to test open-source LLMs, showing accuracy drops with reasoning depth and that small LoRA fine-tuning gains are real but fragile given the small evaluation sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'correct by construction' guarantee attaches to the expression tree, not to the rendered question; the paper admits possible ambiguity in §7 but provides no measurement of how often a rendered (or augmented) question maps to a different computation.","rationale":"The paper's central value proposition is a benchmark where answers are guaranteed correct by construction, at arbitrary scale, with no annotation cost and no generator error rate. For that guarantee to hold, the natural-language question must be an unambiguous encoding of the executable expression tree. Any gap between the tree and the text means the gold answer is correct for a different problem than the one a model solves, which would invalidate both the benchmark scores and the verified CoT traces used for fine-tuning. Section 7 explicitly concedes this risk, but the paper offers no quantitative assessment. The augmentation step in §3.3 makes the gap more likely: injecting 'minor typographical errors' into the text can alter semantically load-bearing tokens, and the claim that all augmentations preserve the computation graph is asserted without evidence. The 'without inheriting a generator's error rate' phrasing is also slightly misleading: a deterministic renderer does not have an LLM-style hallucination rate, but it can have a non-negligible ambiguity rate, and an unmeasured ambiguity rate is a direct threat to the benchmark's validity. This is the most load-bearing concern because it attacks the foundational guarantee rather than secondary claims about LoRA gains or model rankings. The proposed test—independent human parsing of a stratified sample—directly measures the ambiguity rate and would settle whether the 'correct by construction' claim is substantively true for the questions as actually presented to models.","tokens_in":11384,"tokens_out":8618,"duration_ms":75382,"concrete_test":"Sample 200 generated items stratified by depth (0–8) and augmentation variant (clean, typo, paraphrase). For each item, three human annotators with financial background, blind to the gold expression, write the arithmetic computation they understand from the question in a canonical operator notation. Also apply an automated differential check for augmented items: compare the augmented text against the original rendered text and flag any change to operator words, numeric years, or company/concept tokens. Compute the agreement rate between annotator expressions and the gold tree. If fewer than 2% of items are ambiguous or semantically flipped, the concern does not land; if more than 5% are, the 'correct by construction' claim fails and benchmark scores need re-evaluation with filtered items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that V-FiLLM generates QA pairs whose answers are correct by construction. That property holds for the expression tree, but a benchmark question is the natural-language string, and the answer is only correct if the string uniquely determines the tree. Section 3.2 describes a bottom-up renderer that combines operator templates with shared metadata, and the paper's own example in Figure 2 ('difference between total assets scaled by income tax and cost of goods sold scaled by income tax') illustrates how attachment and scope can be ambiguous. Section 7 concedes that mathematical steps 'may occasionally be translated into English in ways that are ambiguous or lack sufficient context,' but no estimate of the ambiguity rate is provided. Section 3.3 adds an unexamined risk: 'minor typographical errors' are injected into questions while asserting the computation graph is preserved. If a typo alters an operator word (e.g., 'maximum' to 'minimum') or a numeric year, the question text denotes a different computation and the stored answer is wrong for the text as read. The phrase 'without inheriting a generator's error rate' is therefore overstated: a deterministic renderer has no stochastic error rate, but it can have a systematic ambiguity rate, and that rate is unmeasured. Until it is shown that human readers independently recover the intended expression from the question at high fidelity, the benchmark's foundational validity property is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces V-FiLLM, a framework that generates financial question-answering items by sampling typed expression trees whose leaves are spreadsheet atoms from synthetic financial data, evaluating the trees symbolically to obtain ground-truth answers, and rendering the trees into natural-language questions. The authors claim these items are correct by construction, scale without annotation cost, and avoid inheriting a generator's error rate, while exposing four independently controllable difficulty axes: computation depth, expression breadth, financial concept complexity, and context size. They evaluate several open-weight models on single-turn and multi-turn versions, report that accuracy declines with depth and under adversarial table perturbations---most dramatically under unit/scale shifts---and find that LoRA fine-tuning on verified chain-of-thought traces improves accuracy on held-out generated questions and yields a small gain on FinQA.","tokens_in":11683,"tokens_out":7029,"duration_ms":59757,"significance":"If the construction is fully validated, V-FiLLM would be a useful evaluation resource for compositional financial reasoning: deterministic symbolic grounding, controllable depth, and machine-checked answers are genuine strengths, and the authors provide code, stored expression-tree metadata, verified chain-of-thought traces, and an external FinQA check. The benchmark also makes falsifiable predictions about model degradation with depth and robustness to input perturbations, which is valuable. The contribution is currently tempered by two load-bearing issues: the correctness guarantee attaches to the expression tree rather than the actual question string, and the text repeatedly describes synthetic data as real filings. Once these are addressed, the resource could be a solid addition to the field, but as written the central claims need revision.","major_comments":[{"comment":"The 'correct by construction' claim holds for the expression tree, not for the rendered natural-language question. The bottom-up renderer can produce attachment and scope ambiguities: the depth-2 example in Figure 2, 'the difference between total assets scaled by income tax and cost of goods sold scaled by income tax', can be read as total assets minus (COGS times income tax) if the second 'scaled by' phrase attaches only to the second noun phrase, whereas the intended tree is (total assets times income tax) minus (COGS times income tax). Because the stored answer is the tree's value, a model solving the alternative parse will be marked incorrect despite following the text. Section 7 concedes that the questions 'may still contain imperfections' and that steps 'may occasionally be translated into English in ways that are ambiguous or lack sufficient context', but no ambiguity rate, no examples, and no human or parser-based verification are provided. The Section 3.3 augmentation that injects 'minor typographical errors' while asserting the computation graph is preserved adds further risk: a substitution such as 'maximum' to 'minimum' or a changed year would alter the denoted computation while the stored answer stays fixed. Consequently, the abstract's claim of generating items 'without inheriting a generator's error rate' is not established; a deterministic renderer can have a systematic, unmeasured ambiguity rate. Please measure the rate at which independent readers or an independent parser recover the intended computation from the question string, or substantially soften the correctness guarantee.","section":"Abstract, Section 3.1, Section 5.1, Tables 2 and 4"},{"comment":"The paper repeatedly calls the evaluation data 'real'. The abstract says the benchmark is 'grounded in real tables', and Section 5.1 and Tables 2 and 4 report results 'over real 10-Q filings' and on 'real documents'. Section 3.1, however, states that both data sources are synthetic: the first 'mimics a typical 10-Q filing' and the second is a 'regularized financial sheet'; the values are generated 'to preserve internal coherence while remaining synthetic'. This is not merely a terminology issue, because Section 5.1 explains performance differences by saying 'real documents have messy layouts, footnotes, and mixed units', attributing to real filings properties that actually belong to the synthetic mimics. Please revise the abstract, Section 5, and the table captions to use 'synthetic 10-Q-style filings' or similar, or add genuine real-world filings to ground the external-validity claims.","section":"Section 4.3, Tables 2-6"},{"comment":"The abstract and Section 4.3 claim the framework 'exposes four independently controllable axes of difficulty'---depth, breadth, financial concept complexity, and context size. The experimental section, however, varies only depth systematically (Table 3). There are no reported experiments that manipulate expression breadth, the frequency or type of derived financial concepts, or the number of companies or rows in the prompt. Section 4.3 also describes 'value scaling' as a difficulty axis, but the unit/scale-shift row in Table 4 is an adversarial perturbation applied to question-irrelevant cells, not the controlled value-scaling manipulation described in that subsection. Either add evaluations for each claimed axis, or revise the claims to match the actually supported axes.","section":"Tables 2-4, Section 5"},{"comment":"The headline numbers are based on 250 mixed-depth questions in Table 2 and 100 questions per cell in Tables 3 and 4, with no confidence intervals, no repeated trials, and no significance tests. In this regime, reported differences of a few percentage points, such as 98.4% versus 98.0%, are within sampling noise, yet the text draws comparative conclusions such as 'Gemma-31B matches or outperforms both GPT-OSS-120B and Qwen3.7-Plus across all conditions' and attributes this to 'specialized training data and robust table parsing mechanisms'. The Section 5.3 hyperparameter sweep is also selected on the same held-out set of 90 problems, as Section 7 acknowledges; Table 5 should therefore be labeled exploratory. Please add confidence intervals or repeated sampling, and soften claims that outrun the statistical evidence.","section":null}],"minor_comments":[{"comment":"The sentence 'See 2 for an example of such trees' should read 'see Figure 2', and the expression '2i for depthi' needs proper mathematical formatting.","section":"Section 3.3"},{"comment":"The final paragraph of Section 3.2 ('Such perturbations enable ... practical financial applications') is repeated verbatim at the end of Section 3.3; remove the duplicate.","section":"Section 4.1"},{"comment":"The sentence beginning 'We convert T into a multi-turn dialogue under the following rules, we convert only trees...' is ungrammatical; rewrite to introduce the conversion rules cleanly.","section":"Section 4.2"},{"comment":"The citation to Mirzadeh et al. (2025) for 'drops of up to 65% from a single such clause' is out of place in a paragraph about tabular perturbations and should be removed or replaced with a directly relevant table-QA robustness citation.","section":"Section 4.4"},{"comment":"Section 4.4 says 'A rank of (r=16), a scaling factor (α=32), and a dropout rate of 0.05 were used', but Section 5.3 reports the best configuration as r=8, α=8, dropout 0.10; please reconcile these statements so the reported recipe is unambiguous.","section":"Abstract"},{"comment":"The abstract contains a typo in 'outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s)'; the stray ', s)' should be removed.","section":"Section 5.2"},{"comment":"The text uses 'pertubations' instead of 'perturbations', and the table captions inconsistently spell '10-Q Fillings' with different capitalization; please correct these throughout.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early-stage technical report: the duplicated text between Sections 3.2 and 3.3, the inconsistent LoRA settings, and the ungrammatical sentence in Section 4.1 suggest incomplete editing. The central issue, however, is not editorial but substantive: the benchmark's validity rests on the mapping from question strings to computation trees, and that mapping is currently unverified. I also note that several evaluated models (e.g., GPT-OSS-120B, Qwen3.7-Plus) may not be identifiable from public sources; if they are proprietary or hypothetical, the paper should state their provenance and access information explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'd say this paper is worth a careful read despite some warts. The core idea—generate financial QA pairs from executable computation trees grounded in synthetic spreadsheets—is a real contribution. Applying the GSM-Symbolic/DyVal style of programmatic generation to finance, with protected derived concepts, multi-turn conversion, and four perturbation families, fills a genuine gap. The deterministic answers and controlled difficulty axes are genuinely useful for isolating model failures in financial reasoning.\n\nWhat the paper does well: the pipeline is clean and the self-verifying answer design is sound at the expression-tree level. The protected-concept rejection step is a thoughtful touch, preventing the benchmark from conflating arithmetic equivalents with named financial abstractions. The multi-turn decomposition is a nice diagnostic that exposes intermediate errors, and the four perturbation families mirror realistic failure modes. The paper is also honest in Section 7, admitting that rendered questions may be ambiguous.\n\nThe main soft spot is exactly what the stress-test hits: the 'correct by construction' guarantee attaches to the expression tree, not to the natural-language question. The Figure 2 example is genuinely ambiguous, and the typo-injection augmentation in Section 3.3 can flip an operator word or a year, making the stored answer wrong for the text as read. The paper admits ambiguity but never measures how often it occurs. For a benchmark whose entire value proposition is verifiable ground truth, that rate is load-bearing. This is fixable—a round-trip check where the rendered question is parsed back to the intended tree, or a human-annotation fidelity sample—but without it the scores have an unknown error floor.\n\nOther issues are smaller. The abstract and Section 5.1 call the data 'real 10-Q filings' while Section 3.1 clearly says synthetic, which will mislead readers. The abstract's 'up to 51%' depth drop doesn't match Table 3, where DeepSeek-v4-Flash drops 58 points from depth 6 to 8. The LoRA results are preliminary (n=90, no significance testing), but the authors flag this themselves, so I won't pile on. And the code URL points to a bare repo with no commit hash, which weakens the reproducibility claim.\n\nThe stress-test is fair but not fatal: this is a limitation the authors admit, not a hidden flaw, and the underlying generation concept holds up. The paper deserves a serious referee who will push on the ambiguity measurement and language consistency rather than a desk reject.\n\nWho's this for? LLM-evaluation researchers and financial ML practitioners who need controlled, scalable benchmarks. I'd send it to review with major revision, but my own verdict after revision would depend on whether they can show that human readers recover the intended computation from the rendered questions with high fidelity.","headline":"A solid, useful benchmark generator for financial tabular reasoning, but the 'correct by construction' claim is stronger than the rendered questions can support until the ambiguity rate is measured.","tokens_in":762,"tokens_out":973,"would_cite":false,"duration_ms":34752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that financial question-answer benchmarks can be generated from executable computation trees with answers correct by construction, no annotation cost, and controlled difficulty, and that current open models degrade…","keywords":["financial reasoning","benchmark generation","computation trees","tabular question answering","chain-of-thought","LoRA fine-tuning","adversarial robustness","synthetic financial data"],"falsifier":"Take a sample of generated questions and ask a blind second process, human annotators or an independent renderer, to reconstruct the expression tree from the question text alone; if any nontrivial fraction of reconstructions differs from the original tree, the correctness-by-construction guarantee fails for those items. A sharper version is to search the generator's output space for two distinct bound expression trees that render to identical question strings with different answers.","tokens_in":11197,"feed_emoji":"📊","tokens_out":5927,"duration_ms":49150,"temperature":0.7,"pith_summary":"This paper introduces V-FiLLM, a framework that generates financial question-answer benchmarks from executable computation trees grounded in synthetic spreadsheets. Because each question is rendered from a tree that is evaluated symbolically, the answer is correct by construction and no model or human annotator is involved in labeling; the authors claim this removes the generator's error rate and makes arbitrarily large benchmark generation possible. The framework exposes four independently controllable difficulty axes: computation depth, expression breadth, financial concept complexity, and context size. Using the benchmark, the authors find that open models' accuracy falls by up to 51 points as reasoning depth grows and by up to 47 points under adversarial numerical perturbations, and that LoRA fine-tuning on verified chain-of-thought traces lifts Qwen3-4B from 81.1% to 85.6% on held-out problems and by 5 points on FinQA. If the construction is sound, V-FiLLM offers a scalable, verifiable evaluation resource for financial reasoning over tables.","feed_headline":"No annotators: financial QA benchmark answers are built in","feed_subtitle":"Generated from executable computation trees, it scales to any depth and shows where reasoning models break.","key_machinery":"The load-bearing object is the typed, executable computation tree: leaves are financial atoms (a spreadsheet cell with its concept, company, year, unit, and quantity type), internal nodes are operations such as addition, subtraction, multiplication, ratio, growth, minimum, maximum, and average, and a type-aware sampler rejects incompatible compositions. The tree is evaluated symbolically to produce the ground-truth answer and rendered bottom-up into natural-language questions, so the answer is guaranteed correct for the intended computation. Protected derived financial concepts carry hidden formula trees, and a rejection step prevents a raw arithmetic composition from being mistaken for a named concept. This tree structure is also what makes depth, breadth, value scaling, and multi-turn decomposition directly controllable.","core_discovery":"The paper's central claim is that financial reasoning benchmarks can be generated deterministically rather than collected from human annotation: synthetic financial spreadsheets are decomposed into typed atoms, composed into typed binary expression trees, and rendered into natural-language questions, with the ground truth obtained by symbolically evaluating the tree. This makes answers correct by construction, at arbitrary scale, without inheriting a generator's error rate. The same tree structure gives explicit control over reasoning depth, expression breadth through balanced trees, the presence of named derived financial concepts such as gross profit and operating income, and context size. The empirical payoff is that accuracy degrades sharply with depth and with table perturbations, with unit and scale shifts the most destructive on realistic 10-Q filings, and that multi-turn decomposition and verified-trace LoRA fine-tuning both recover a substantial share of the lost performance.","pith_inferences":["An extension the paper leaves implicit is that the same tree-rendering pipeline transfers to other structured domains, such as medical records or scientific tables, by swapping the financial concept library; the correctness-by-construction guarantee would carry over.","The unit and scale fragility result implies models may be matching digit patterns rather than reading unit semantics; the benchmark's value-scaling axis already provides a direct way to test this.","Because every tree node carries ground truth, the verified traces could also support process-supervision or verifier-guided decoding; the paper only demonstrates the LoRA use.","A back-rendering check, recovering the expression tree from the question text and comparing it with the original tree, would quantify the ambiguity that the paper's limitations section concedes."],"forward_implications":["Because no human labels or model outputs enter answer generation, the benchmark can be scaled to arbitrary depth and breadth distributions at near-zero marginal cost, enabling fine-grained difficulty sweeps.","Per-depth accuracy curves provide a direct diagnostic of where compositional reasoning fails, isolating intermediate arithmetic errors from planning errors.","The multi-turn version, which converts each internal node into a conversational turn, exposes intermediate failures that single-turn accuracy hides and benefits the weakest models most.","The adversarial perturbation suite, with ground truth invariant by construction, cleanly attributes accuracy drops to model fragility rather than label shift.","LoRA fine-tuning on verified chain-of-thought traces gives a small but consistent accuracy gain and transfers to FinQA, indicating that verified synthetic traces are useful training data."],"supporting_citations":[{"why":"FinQA is the external dataset used to test whether LoRA gains transfer beyond the generated benchmark, and it is the main human-annotated baseline the framework contrasts with.","marker":"(Chen et al., 2022a)"},{"why":"Supplies the LoRA method used for parameter-efficient fine-tuning on verified chain-of-thought traces.","marker":"(Hu et al., 2021)"},{"why":"Provides prior evidence that reasoning depth and simple perturbations degrade LLM math performance, motivating the depth and perturbation axes.","marker":"(Mirzadeh et al., 2025)"},{"why":"Documents transformer degradation with compositional depth, the phenomenon the benchmark is designed to measure.","marker":"(Dziri et al., 2023)"},{"why":"Shows prompt-format sensitivity, motivating the linguistic augmentation and robustness evaluations.","marker":"(Sclar et al., 2023)"},{"why":"Provides the invariance-testing paradigm behind the perturbation and augmentation suites.","marker":"(Ribeiro et al., 2020)"},{"why":"Least-to-most decomposition motivates the multi-turn extension and explains why turn-by-turn questioning helps.","marker":"(Zhou et al., 2023)"},{"why":"Table QA adversarial perturbation study that the robustness protocol follows.","marker":"(Zhao et al., 2023)"}],"fun_headline_variants":["Financial QA benchmark built from computation trees, no annotators","Reasoning depth costs 51% accuracy in new financial benchmark","Verified-trace LoRA lifts financial QA accuracy to 85.6%","Zero-annotation financial reasoning benchmark scales via trees","Why financial LLM reasoning breaks: depth and perturbations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the bottom-up renderer turns each computation tree into a natural-language question whose wording unambiguously picks out that computation; if a question can be read as a different calculation, the 'correct by construction' answer is no longer the answer to the question being posed.","fun_headline_variants_meta":{"raw":{"variants":["Financial QA benchmark built from computation trees, no annotators","Reasoning depth costs 51% accuracy in new financial benchmark","Verified-trace LoRA lifts financial QA accuracy to 85.6%","Zero-annotation financial reasoning benchmark scales via trees","Why financial LLM reasoning breaks: depth and perturbations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2883,"prompt_tokens":922,"completion_tokens":1961,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1878}},"tokens_in":538,"tokens_out":1961,"duration_ms":11772,"temperature":1.0,"reasoning_tokens":1878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:22:53.263651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of generated questions and ask a blind second process, human annotators or an independent renderer, to reconstruct the expression tree from the question text alone; if any nontrivial fraction of reconstructions differs from the original tree, the correctness-by-construction guarantee fails for those items. A sharper version is to search the generator's output space for two distinct bound expression trees that render to identical question strings with different answers.","supporting_citations":[],"review_version":1}