{"id":"3262636a-01ba-4951-a356-7aa948d35e27","arxiv_id":"2608.11381","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Specialist prompt decomposition improves numerical financial tasks but not judgment tasks on a frontier LLM, whereas GRPO post-training of a smaller model improves judgment tasks with transfer to unseen firms.","lead":"This paper tests whether splitting a financial AI analyst into specialized sub-agents improves performance, and finds that it helps with number-based tasks but not with integrative judgment tasks. The authors then use reinforcement learning to train a smaller model to judge better, and report gains that transfer to companies and legal regimes the model had not seen.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Division-of-labor claim rests on an untested model-family interaction: decomposition is shown only on Claude, RL only on Qwen, so the task-dependent principle is not identified.","rationale":"Reader's weakest assumption matches my primary concern: the design principle is inferred from different model families. I checked the full text for a same-model cell: Section 3.4 says the design separates prompt-level and parameter-level effects, but the frontier comparison is Claude-only and the RL comparison is Qwen-only. Section 5.4's final transfer check is still Qwen-only and even shows large model-specific floor effects (T1/T3 at 0.0-1.8% for Qwen vs 84-100% for Claude), which underscores that model family is a first-order variable. The development-split selection issue is real but secondary, because held-out transfer and the final primary benchmark provide some out-of-sample support for the RL effect. The model-family interaction, by contrast, is a logical gap in the central claim: the two interventions are never compared within a model, and the conclusion requires a cross-model invariance that is neither tested nor argued. A Qwen decomposition run would settle it cheaply. Since the paper already labels this as future work and the reader already made acceptance conditional, I do not move the verdict.","tokens_in":11249,"tokens_out":4571,"duration_ms":41729,"concrete_test":"Run the identical three-condition frontier protocol (monolithic, full-framework, specialist) from Section 4.2 on Qwen3.5-9B zero-shot, holding source evidence, task instructions, output schemas, scoring, and K=4 dispatch pairing fixed, on the 19-firm/95-cell primary benchmark. Compare Snum and Sjudg for Qwen-Mono vs Qwen-Spec using the same paired sign-flip test. If Qwen-Spec reproduces a numerical gain near +15.8 points with no judgment gain, the prompt-level half of the division-of-labor principle is model-invariant; if the numerical gain disappears or judgment also improves, the central claim is confounded by model family and should be weakened to a conjunction of two model-specific results. This is the cheapest decisive cell because it requires no additional RL training and directly fills the missing interaction between intervention type and model family.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 measures prompt decomposition only on Claude Opus 4.8, and Section 4.4 measures GRPO post-training only on Qwen3.5-9B. The abstract and Section 7 then conclude that prompt-level decomposition improves modular numerical execution while parameter adaptation improves integrative judgment. That conclusion requires the task-dependent pattern to be invariant across base models, or at least requires a same-model comparison of the two interventions. No such cell exists: there is no Qwen monolithic-vs-specialist decomposition run, and no second-model RL run; the paper explicitly lists a frontier-model comparison on held-out splits as future work. The observed pattern is therefore consistent with a different explanation: Claude's judgment tasks may be insensitive to prompt restructuring in this format, while Qwen's zero-shot judgment policy is weak (T5 at 37.2% on dev and 38.1% on unseen firms) and easily moved by any reward signal. If that is the case, the 'division of labor' is a property of the two specific checkpoints, not of the intervention types. The paper's own limitations section acknowledges one frontier model family and one 9B open model, so this is a flagged missing support rather than an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Larix, an eight-specialist agent layer mapped to a 16-lens framework for European listed real estate, and evaluates two forms of LLM specialization on a five-task benchmark over 19 continental firms (plus six Swiss firms in auxiliary analyses). With Claude Opus 4.8 held fixed, lens-aligned specialist prompting raises the numerical aggregate (T1/T3/T6) from 78.9% to 94.7% (paired McNemar p=0.012) while leaving the judgment aggregate (T2/T5) at 47.4% in the primary dispatch; a monolithic prompt with the full 16-lens framework does not reproduce the numerical gain. Separately, GRPO post-training of Qwen3.5-9B with the deterministic rubric as reward raises the development-split overall score by 12.0 points and the judgment aggregate by 14.2 points, with positive transfer on held-out firms, wrappers, and later periods. The paper concludes that prompt-level decomposition improves modular numerical execution while parameter-level adaptation improves integrative financial judgment.","tokens_in":11434,"tokens_out":8175,"duration_ms":65959,"significance":"If the conclusion holds, the paper provides a controlled same-model demonstration that prompt decomposition has a task-dependent effect, a useful full-framework control, and an anti-memorization transfer design for RL in a domain setting. The deterministic scorer, precommitted ground truth with expert-blinded tuples, frozen-template dispatches, and three disjoint out-of-distribution splits are genuine strengths, as is the planned release of the benchmark asset with per-cell provenance. The main limitation is that the two sides of the division-of-labor claim are established on different base models, so the headline principle is an inference across model families rather than a demonstrated two-by-two comparison; this gap is candidly acknowledged in the limitations section but remains load-bearing for the central claim.","major_comments":[{"comment":"Decomposition is evaluated only on Claude Opus 4.8 and RL post-training only on Qwen3.5-9B. The abstract's concluding sentence ('Prompt-level decomposition thus improves modular numerical execution, whereas targeted parameter adaptation improves integrative financial judgment') and Section 7 require the task-dependent pattern to be invariant to base model. No same-model cell tests decomposition on Qwen or RL on Claude; Table 2's own note says the Qwen columns are not directly comparable to the 19-firm cohort. The observed pattern is also consistent with the alternative that Claude's judgment tasks are insensitive to prompt restructuring while Qwen's zero-shot judgment policy is weak (T5 at 37.2% on dev and 38.1% on unseen firms) and improves with almost any reward signal. Please add at least one within-model crossing, for example specialist versus monolithic prompting on Qwen, or explicitly weaken the division-of-labor claim to a conjecture.","section":"§4.2, §4.4, §7; Table 2"},{"comment":"The Claude-Full T2/T5/T6 cells come from a frozen-template re-dispatch dated 2026-08-09, whereas the Claude-Mono and Claude-Spec cells come from the primary runs. The full-framework control is load-bearing for the argument that framework disclosure does not explain the numerical gain and for the symmetric T1/T2 pattern. A later re-dispatch can differ due to API-side model updates, decoding nondeterminism, or template drift. Please either re-dispatch all three frontier conditions in the same batch or report the re-dispatch as a separate control and do not pool it with the primary comparison without a stability check.","section":"Table 2 note; §5.2"},{"comment":"The 95% confidence interval [5.1,26.4] and p=0.012 for the numerical aggregate come from a paired McNemar test that pools 57 task-firm cells as independent units. Since the benchmark contains 19 firms and three tasks, responses from the same firm are likely correlated; the firm-clustered sign-flip test reported (p=0.001) is for the suite-level 95 instance differences, not for the numerical aggregate. Please report a firm-clustered test for Snum specifically, or state the intra-firm correlation, so the 'reliably improves' claim is not carried by an unclustered p-value.","section":"§5.1, Eq. (11)-(12)"},{"comment":"The abstract and conclusion headline the development-split gain (12.0 points overall; +14.2 on judgment), but the development split is used for checkpoint selection and early stopping, and the evaluated step-20 checkpoint was selected on that split. Section 5.4's out-of-distribution results support transfer, so this is not fatal, but the headline numbers carry selection bias. Please either flag the development-split numbers as selection-biased throughout the abstract and conclusion, or lead with the held-out-firm and held-out-wrapper gains.","section":"§4.4, §5.3, §7"}],"minor_comments":[{"comment":"The text 'four German AGs, 1 one Italian SIIQ' contains a duplicated numeral '1'; it should read 'one Italian SIIQ'.","section":"§4.1"},{"comment":"The abstract says decomposition 'can reduce' judgment performance, but the aggregate judgment effect across dispatches ranges from 0.0 to +10.5 points; the reduction is at the task level (T2, -5.2 points in the primary dispatch). Please make this distinction explicit.","section":"§5.1"},{"comment":"The caption does not describe the axes or the rolling window; please add axis labels, the number of steps shown, and the deduplication rule for the in-training validation curve.","section":"Figure 1"},{"comment":"The report that the training pod's storage was lost and that only the step-20 checkpoint was preserved is a reproducibility concern; please include the full training configuration (learning rate, LoRA alpha, sequence length, rollout batch details) in an appendix so the run can be recreated.","section":"§4.4"},{"comment":"The statistical protocol states that condition labels are permuted jointly within firm over 2^19 assignments; please state how ties are handled in the sign-flip test and whether the reported p-values include a correction for multiple dispatches.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is candid about its limitations, which is a strength. The main gap is the missing same-model crossing between prompt decomposition and RL post-training; given the small benchmark (19 firms, five tasks), I would not reject the paper, but the central division-of-labor claim needs either a within-model comparison or a substantially weakened framing. The reference to the authors' prior workflow paper [16] is appropriate and does not itself raise novelty concerns. The benchmark release is valuable and should be preserved through revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. This paper has the cleanest controlled comparison of specialist decomposition I've seen in the financial-LLM-agent literature: same model, same evidence, same schema, plus a full-framework monolithic control. That alone justifies a look. But the abstract's division-of-labor claim — decomposition for numerical tasks, parameter adaptation for judgment — is a conjunction of results from two different model families: decomposition was tested only on Claude Opus 4.8, RL post-training only on Qwen3.5-9B. There is no same-model cell, and the paper lists a frontier-model comparison as future work. So the principle as stated is not identified.\n\nWhat is actually new and good: the same-model comparison isolates decomposition from the confounds that plague prior multi-agent work (extra instructions, different models). The numerical gain (78.9% to 94.7%, paired p=0.012) is credible, and the full-framework control scoring worse on T1 shows the gain isn't just from adding domain text. The RL section is careful too: deterministic rubric as reward, three anti-memorization splits, reward telemetry showing no collapse on the breach-classification term, and greedy-decoding sensitivity checks. The appendix discloses harness-hardening failures (field-request alignment, rubric sub-classification) with unusual candor.\n\nSoft spots, in proportion. The headline RL judgment gain (+14.2 points) is measured on the development split that was used to select the step-20 checkpoint — that's selection on the evaluation metric. The held-out transfer numbers are encouraging (T5 +40.4 on unseen firms) but noisier, and T5 is flat on unseen wrappers. Second, the central design principle crosses model families without a within-family comparison; the observed pattern could be a property of the specific checkpoints (Claude's judgment being insensitive to prompt scope, Qwen's judgment being weak and easily improved by any reward signal). The paper acknowledges this in limitations, so it's a missing support rather than a contradiction. Minor: the released benchmark asset is not linked, and some Claude-Full cells come from a later frozen-template re-dispatch rather than the same run.\n\nWho this is for: people building financial LLM agents, and anyone studying prompt-level versus weight-level specialization. The 19-firm benchmark is small, but the evaluation template is worth adopting. I'd send it to a serious referee. The request would be: either add a same-model comparison (even one cell: Qwen-Mono vs Qwen-Spec, or Claude zero-shot vs Claude after similar post-training) or soften the conclusion to an explicit hypothesis. As written, the evidence supports \"scoped prompts help localized numerical extraction\" and \"RL post-training improves judgment on a 9B model,\" but not the strong cross-model principle.","headline":"A careful empirical study with a genuinely clean decomposition control, but the headline division-of-labor claim crosses model families without a same-model comparison and the RL headline sits on the checkpoint-selection split.","tokens_in":11994,"tokens_out":4020,"would_cite":true,"duration_ms":36394,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Specialist prompting lifts numerical financial tasks; reinforcement post-training lifts judgment tasks.","keywords":["financial LLM evaluation","specialist agent decomposition","reinforcement learning post-training","GRPO","European listed real estate","regime-aware analysis","structured output scoring"],"falsifier":"Apply the same lens-aligned specialist prompts to the 9B model before post-training: if its judgment aggregate rises by roughly 14 points with no RL, or if the post-trained checkpoint's judgment gain disappears when the same GRPO recipe is run on a different 9B base model, the paper's task-dependent division-of-labor claim is unsupported.","tokens_in":10989,"feed_emoji":"📊","tokens_out":8324,"duration_ms":78025,"temperature":0.7,"pith_summary":"This paper asks whether the two distinct demands of financial analysis—localized numerical computation and integrative judgment—benefit from the same kind of LLM specialization. On a five-task, 19-firm benchmark of European listed real estate spanning seven legal and reporting regimes, lens-aligned specialist prompts on a frontier model raise the numerical-task aggregate from 78.9% to 94.7% while leaving the judgment aggregate at 47.4% in the primary dispatch. A monolithic control given the full 16-lens framework does not reproduce the numerical gain, so the improvement is attributed to scoped lens assignment rather than extra framework text. Separately, GRPO post-training of a 9B open model raises its judgment aggregate by 14.2 points on the development split, with positive transfer to unseen firms, wrappers, and periods. The paper's central claim is that prompt-level decomposition improves modular numerical execution, whereas targeted parameter adaptation improves integrative financial judgment.","feed_headline":"Specialist prompts fix numbers; post-training fixes judgment","feed_subtitle":"On 19 real-estate firms: scoped prompts +15.8 on numbers; RL +14.2 on judgment","key_machinery":"The load-bearing device is a paired same-model intervention. On the frontier model, the only change across conditions is whether the 16-lens framework is presented as one monolithic prompt or as a task-routed specialist prompt, with evidence, output schema, and scoring fixed. On the small model, the only change is GRPO post-training—group-relative policy optimization—against the same deterministic rubric that scores evaluation, so the reward is task-aligned rather than a model-based judge. The benchmark separates tasks into numerical (localized extraction and calculation) and judgment (multi-disclosure reconciliation), and the same deterministic scorer is used for the frontier comparison, the small-model baseline, RL reward computation, and held-out evaluation. That shared interface is what lets the paper attribute numerical gains to scoped prompting and judgment gains to parameter adaptation rather than to scoring or prompt artifacts.","core_discovery":"Using a 16-lens analytical framework mapped to eight lens-aligned specialists, the paper evaluates a fixed frontier model under three prompting conditions—generic monolith, full-framework monolith, and routed specialist prompt—with identical source evidence, output schemas, instructions, and scoring. Specialist decomposition improves the numerical aggregate (regime-specific metric, implied cap rate, payout-regime classification) by 15.8 percentage points, stable across four frozen-template dispatches, while the judgment aggregate (reconciliation adjustment, covenant stress) is unchanged in the primary dispatch and gains only inconsistently across repeats. Giving the complete framework to a single monolithic prompt instead helps the judgment task and hurts extraction, showing the numerical gain is tied to scoped assignment rather than framework disclosure. On the same task interface, GRPO post-training of a 9B open model with the deterministic rubric as reward raises the development-split score by 12.0 points and the judgment aggregate by 14.2 points, and transfers positively to unseen firms (+15.2 overall; +40.4 on covenant stress), unseen regulatory wrappers (+4.3), and later periods (+2.5). Together these results support a task-dependent design principle: use decomposition for localized, verifiable numerical operations and parameter-level post-training for integrative financial judgment.","pith_inferences":["If the task-dependent pattern is model-invariant, the same division of labor should appear when specialist decomposition is applied to the small model and GRPO to the frontier model; this cross-over experiment is the natural next test the paper leaves implicit.","The large transfer gain on covenant stress (+40.4 points on unseen firms) suggests that multi-disclosure judgment tasks may be the highest-value target for RL post-training in regulated finance, since they are where zero-shot models are weakest.","The harness-hardening report that prompt–scorer misalignment can create zero-by-construction cells implies that published gains on structured financial tasks should be read alongside the exact schema instructions, a caution that extends beyond this benchmark.","Because the evaluated layer excludes downstream synthesis and position sizing, the division-of-labor claim is scoped to single-specialist invocation; a message-passing pipeline could behave differently if specialists exchange evidence."],"forward_implications":["For financial-analysis systems, routing a task to a single lens-aligned specialist prompt is a cheap way to improve regime-aware extraction and calculation without changing the model.","Integrative judgment tasks, such as covenant stress assessment, should not be expected to improve from narrower prompts; the paper's evidence says they may stay flat or decline under scoped decomposition.","Post-training a smaller model with the evaluation rubric as reward can improve judgment tasks that prompt decomposition misses, and the gains generalize to firms, wrappers, and periods not seen in training.","Because the deterministic scorer is the RL reward, post-training and evaluation are aligned; a model that maximizes the rubric is optimized for the same structured outputs the benchmark scores.","The controlled comparison indicates that the gains come from the form of specialization—prompt scope versus parameter adaptation—not from additional domain instructions alone."],"supporting_citations":[{"why":"Supplies the GRPO group-relative policy-optimization objective used for the RL post-training runs.","marker":"[3]"},{"why":"Defines the EPRA best-practice reporting conventions that ground the regime-specific tasks and the answer key.","marker":"[4]"},{"why":"Identifies the 9B open base model checkpoint that receives the LoRA and GRPO post-training.","marker":"[13]"},{"why":"Provides the RL training and rollout infrastructure used to execute the GRPO post-training.","marker":"[15]"},{"why":"Supplies the evidence that FFO carries information beyond GAAP net income, which the operating-metric task encodes.","marker":"[17]"},{"why":"Provides the chain-of-thought decomposition baseline that the specialist condition must beat.","marker":"[18]"}],"fun_headline_variants":["Scoped prompts boost numbers; RL boosts judgment","Decomposition for numbers, RL for judgment","Specialists fix numeric tasks, RL fixes judgment","Numbers need scoped prompts; judgment needs RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's division-of-labor conclusion assumes that the pattern—prompt decomposition helping numerics but not judgment, RL post-training helping judgment—is a property of the task types rather than an accident of the two different base models used for the two interventions.","fun_headline_variants_meta":{"raw":{"variants":["Scoped prompts boost numbers; RL boosts judgment","Decomposition for numbers, RL for judgment","Specialists fix numeric tasks, RL fixes judgment","Numbers need scoped prompts; judgment needs RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00038,"raw_usage":{"total_tokens":2064,"prompt_tokens":1038,"completion_tokens":1026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":968}},"tokens_in":654,"tokens_out":1026,"duration_ms":10901,"temperature":1.0,"reasoning_tokens":968,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:42.999720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same lens-aligned specialist prompts to the 9B model before post-training: if its judgment aggregate rises by roughly 14 points with no RL, or if the post-trained checkpoint's judgment gain disappears when the same GRPO recipe is run on a different 9B base model, the paper's task-dependent division-of-labor claim is unsupported.","supporting_citations":[{"cited_title":"DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the GRPO group-relative policy-optimization objective used for the RL post-training runs."},{"cited_title":"EPRA best practices recommendations (BPR) guidelines","cited_arxiv_id":null,"evidence_quote":"Defines the EPRA best-practice reporting conventions that ground the regime-specific tasks and the answer key."},{"cited_title":"Qwen3.5-9B (hugging face model card)","cited_arxiv_id":null,"evidence_quote":"Identifies the 9B open base model checkpoint that receives the LoRA and GRPO post-training."},{"cited_title":"Sheng, S","cited_arxiv_id":null,"evidence_quote":"Provides the RL training and rollout infrastructure used to execute the GRPO post-training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evidence that FFO carries information beyond GAAP net income, which the operating-metric task encodes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought decomposition baseline that the specialist condition must beat."}],"review_version":1}