{"id":"aab7b78a-985d-4361-82a1-2024f57eae4a","arxiv_id":"2504.14233","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Decomposing a report template into per-section subqueries yields more detailed and better-scored financial reports than a multi-agent system that uses the full template at once.","lead":"This paper compares two ways to generate structured financial reports from earnings call transcripts: an agentic multi-agent system and a decomposed prompt-chaining approach. The decomposed approach produced longer, more detailed reports and scored higher on both LLM-based and reference-based metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Financial result rests on unvalidated length-sensitive LLM metrics; the only reference-based check (SumIPCC) has n=7 with no significance test, and Table 1 contradicts the 'all metrics' claim.","rationale":"The reader's weakest assumption correctly identifies that the financial evaluation is not validated and may be confounded by output length. I agree that this is the main soft spot. However, I would add two concrete aggravating details. First, the SumIPCC 'independent' support is much weaker than it appears: it uses only 7 reports and no significance test, so the abstract's claim of statistical significance in both scenarios is not fully supported. Second, the paper's own Table 1 contains a cell (Financial Takeaways DE: 0.23 for DecomposedIR vs 0.26 for AgenticIR) that contradicts the text's claim of consistent superiority across all metrics. This is either a typo or a sign that the aggregate significance hides per-characteristic failures, and either way it needs correction. The paper does have genuine independent support in the SumIPCC reference-based metrics, and the direction of the effect is plausible and consistent with prior prompt-chaining results. The right outcome is therefore to require the missing human validation and the corrected, document-level statistical reporting rather than to reject the work outright. The conditional verdict stands as is, so no adjustment is needed.","tokens_in":10260,"tokens_out":8317,"duration_ms":79644,"concrete_test":"Select a stratified random sample of 20 company-quarter pairs from the financial dataset. Blind two finance experts to system identity and have them rate each report on the four characteristics. Also rescore DecomposedIR after truncating each report to the sentence count of its AgenticIR counterpart, re-running DecompEval and G-Eval on the truncated versions. If experts do not rank DecomposedIR above AgenticIR, or if truncation removes the metric gap, the financial half of the central claim is an artifact. Report paired 95% confidence intervals and per-characteristic p-values instead of a single aggregate chi-square.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DecomposedIR beats AgenticIR in both scenarios depends mainly on the financial evaluation, which uses reference-free LLM metrics (DecompEval and G-Eval) with no human calibration. DecomposedIR produces 77.7 sentences per report versus 25.0 for AgenticIR (Table 2), and because DecompEval aggregates sentence-level adherence judgments while G-Eval assigns holistic scores that are sensitive to verbosity, the reported 27% average gain may reflect output volume rather than template coverage. The same model family (GPT-4o-mini) is used for generation and evaluation, and no expert validation is provided for the four characteristics. The cross-domain SumIPCC result uses reference-based ROUGE/BERTScore and agrees in direction, which is genuine support, but it is computed on only 7 reorganized reports with no significance test or confidence interval, so it cannot independently establish 'statistically significant in both scenarios.' Additionally, Table 1 shows DecomposedIR scoring lower than AgenticIR on Financial Takeaways DE (0.23 vs 0.26), which directly contradicts the text's statement that DecomposedIR 'consistently outperform[s] AgenticIR methods in all metrics'; either the table has a reporting error or the per-characteristic statistical claim is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two LLM-based workflows for generating templated financial reports from earnings releases: an AutoGen-based multi-agent framework (AgenticIR) and a decomposed prompt-chaining framework (DecomposedIR) in which each template section becomes a subquery answered with retrieved transcript chunks and financial statements. Experiments on 74 earnings documents from five semiconductor companies use reference-free DecompEval and G-Eval over four characteristics plus readability metrics; a second scenario uses the SumIPCC climate dataset with ROUGE/BERTScore against expert-written summaries. The authors report that DecomposedIR statistically significantly outperforms AgenticIR in both scenarios and that self-reflection improves metric scores while increasing text complexity.","tokens_in":10536,"tokens_out":4233,"duration_ms":37386,"significance":"The comparison addresses a practical question in applied LLM system design, and the paper is transparent about data and code availability. It also includes a cross-domain, reference-based check that goes in the same direction as the main result, which is a genuine strength. However, the headline statistical claim currently rests on unvalidated, length-sensitive LLM evaluation in the financial domain and on a very small reference-based dataset with no significance testing; these issues must be resolved before the comparative conclusion can be accepted.","major_comments":[{"comment":"DecompEval aggregates sentence-level judgments, and DecomposedIR produces roughly three times more sentences (77.72 vs 25.01; 96.07 vs 31.82 with self-reflection). A longer report can therefore accumulate a higher DecompEval score even if its sentence-level density of relevant content is identical or worse. G-Eval's holistic scores are also sensitive to verbosity and detail volume. The paper does not report precision-style metrics such as the fraction of sentences judged on-topic, length-controlled comparisons, or human judgments, so the reported 27% average gain cannot be attributed to template coverage rather than output volume.","section":"4.1, Tables 1 and 2"},{"comment":"The claim of statistical significance is not reproducible: Pearson's chi-square test is invoked with p<0.05, but no test statistic, degrees of freedom, p-value per characteristic, or unit of analysis is given. The reported numbers are average scores rather than raw counts, so it is unclear what contingency table was tested. Please specify the test construction and provide per-characteristic statistics; otherwise this central claim is unsupported.","section":"4.1, statistical test"},{"comment":"The cross-domain evidence comprises only seven reorganized reports, and no significance test, confidence interval, or variance measure is reported. With n=7 and differences of 0.08 in ROUGE-1 and 0.04 in BERTScore, the statement that DecomposedIR statistically significantly outperforms AgenticIR in both scenarios is not established. The reorganization of 140 summaries into seven multi-section reports also creates dependencies among evaluation units; please justify this aggregation and report report-level variance.","section":"4.2, Table 3"},{"comment":"The financial-domain evaluation uses GPT-4o-mini both as the report generator and as the DecompEval/G-Eval judge, and the four characteristics are not validated against expert human annotation. This self-evaluation loop is particularly problematic because the judge may systematically favor the longer, more verbose outputs of DecomposedIR. At minimum, calibrate the LLM metrics on a small set of expert-rated reports or report inter-judge agreement with human financial analysts.","section":"4.1, evaluation metrics"},{"comment":"The text states that DecomposedIR methods consistently outperform AgenticIR methods in all metrics, but Table 1 shows Financial Takeaways DE of 0.23 for DecomposedIR versus 0.26 for AgenticIR, and 0.27 for AgenticIR with self-reflection. This internal inconsistency either indicates a reporting error or shows that the per-characteristic advantage is not universal; it should be corrected and the claims qualified.","section":"4.1, Table 1"}],"minor_comments":[{"comment":"The word 'dataet' should be corrected to 'dataset'.","section":"4.2"},{"comment":"AutoGen is cited twice with inconsistent reference details; please consolidate into a single consistent citation.","section":"References [32] and [33]"},{"comment":"The phrase 'apple-to-apple comparison' should be 'apples-to-apples comparison'.","section":"4.1"},{"comment":"Readability differences are reported without statistical tests; please state whether the observed differences are meaningful beyond sampling variation.","section":"4.1, Table 2"},{"comment":"The phrase 'statistically significantly outperforms' is used for both scenarios, but no significance test is described for the SumIPCC scenario; align the terminology with the analyses actually reported.","section":"Abstract and Conclusion"},{"comment":"The retrieval depth n=3 is fixed without sensitivity analysis; since both methods depend on the retrieved chunks, a brief robustness check would strengthen the comparison.","section":"4, experimental setup"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a competently written short paper with a useful practical question, but the experimental support for the headline claim is thinner than the abstract suggests. The reference-based SumIPCC result is the most trustworthy evidence but is too small, and the financial result needs human calibration and length control. I recommend major revision rather than rejection because the comparison framework is sound and the issues are addressable with additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the comparison is useful and the direction probably holds, but the paper's central claim—that DecomposedIR statistically significantly outperforms AgenticIR in both scenarios—is not established by the evidence as presented. The financial-domain numbers are inflated by a length confound, and Table 1 itself undercuts the 'all metrics' wording.\n\nWhat is actually new: this is the first head-to-head of an AutoGen-style agentic IR loop against decomposed prompt chaining for template-based financial report generation. That is a real, practical question. The setup is reasonable: same embedding model, same top-n retrieval, same generator model, self-reflection variants added. They also try a cross-domain check on SumIPCC with ROUGE/BERTScore, which is the right instinct.\n\nThe soft spots. In the financial dataset, DecomposedIR outputs ~78 sentences per report on average versus ~25 for AgenticIR. DecompEval sums sentence-level adherence judgments, so longer output mechanically accumulates coverage; G-Eval is sensitive to verbosity as well. Generation and evaluation both use GPT-4o-mini, so this is a self-evaluation loop. The Pearson chi-square test is underspecified—no unit of analysis, no per-characteristic breakdown. And Table 1 shows AgenticIR actually beating DecomposedIR on Financial Takeaways DE (0.26 vs 0.23), so the sentence 'consistently outperform AgenticIR methods in all metrics' is simply false as written. The SumIPCC result points the same way and uses reference-based metrics, which is genuine supporting evidence, but with n=7 and no significance test it cannot carry the 'statistically significant' claim.\n\nWho this is for: practitioners choosing between an agent framework and a prompt-chaining pipeline. The paper is a pilot, not a definitive benchmark. It deserves a serious referee—the question is timely and the comparison structure is honest—but it needs major revision: length-controlled or length-conditional analysis, human or at least independent-model evaluation on the financial set, a properly described statistical test, and a corrected Table 1/text alignment.\n\nRecommendation: send it to peer review rather than desk-reject, with clear instructions to address the length confound and the Table 1 inconsistency.","headline":"DecomposedIR likely beats AgenticIR for template-based report generation, but the headline significance claim rests on a length-confounded, self-evaluated metric and a table that contradicts it.","tokens_in":11043,"tokens_out":2477,"would_cite":false,"duration_ms":20990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that DecomposedIR, which breaks a report template into a chain of sub-questions answered with retrieved evidence, produces broader and more detailed templated reports than the multi-agent AgenticIR, with statistical…","keywords":["template-based report generation","financial report generation","earnings call summarization","agentic information retrieval","decomposed prompting","prompt chaining","reference-free evaluation","large language models"],"falsifier":"Have financial analysts rate a matched sample of AgenticIR and DecomposedIR reports for the same earnings calls, or recompute DecompEval on length-matched subsamples of the reports; if human preference or length-matched scores do not reproduce the reported 27% advantage, the central claim fails.","tokens_in":10094,"feed_emoji":"📊","tokens_out":8327,"duration_ms":64528,"temperature":0.7,"pith_summary":"This paper asks which of two LLM workflows writes better reports when the output must follow a fixed multi-section template: an agentic system in which several agents collaborate with the whole template in view, or a decomposed system in which the template is broken into sub-questions and answered step by step with retrieved evidence. The authors build both pipelines for quarterly earnings releases from five semiconductor companies and evaluate the generated reports on four financial characteristics: financial takeaways, financial context, reasoning correctness, and management expectation. They report that the decomposed approach, DecomposedIR, statistically significantly outperforms the agentic approach on all four characteristics with p < 0.05, and that the same pattern holds on the SumIPCC climate-report dataset with expert-written references, where DecomposedIR beats AgenticIR by 33% in ROUGE-1 and 6.3% in BERTScore. The conclusion is that explicit template-to-query decomposition helps an LLM bridge the gap between unstructured source text and a structured template more reliably than autonomous agent collaboration. This matters for practitioners deciding whether to invest in multi-agent orchestration or use simpler prompt chaining for structured reporting.","feed_headline":"Decomposed prompts beat multi-agent teams for financial reports","feed_subtitle":"Splitting a report template into sub-questions yields broader, more detailed coverage than a six-agent pipeline.","key_machinery":"The mechanism that carries the argument is template-to-subquery decomposition via prompt chaining. DecomposedIR takes the analyst-defined report template, has an LLM convert each section into a specific sub-question, retrieves the top three transcript chunks plus structured financial statements for each sub-question, answers each one, and then aggregates the answers into a final report aligned with the template. AgenticIR, by contrast, presents the entire template to a team of agents equipped with task-decomposition and retrieval-generation functions, letting the agents choose how to split the work. The load-bearing difference is that decomposed sub-queries force the model to address every template section explicitly, whereas the agentic approach depends on the agents' autonomous decomposition staying aligned with the template.","core_discovery":"On the paper's own terms, the central discovery is that DecomposedIR outperforms AgenticIR for template-based financial report generation across all four key characteristics, on average by 27% with p < 0.05, and that the advantage transfers to a different domain: on SumIPCC, DecomposedIR scores 0.32 versus 0.24 in ROUGE-1 and 0.67 versus 0.63 in BERTScore. The authors interpret this as evidence that the agentic framework's flexible, autonomous division of labour is less able to align generated content with a structured template than a prompt-chaining workflow that reframes each template section as a retrieval-augmented question. They also find that adding a self-reflection step improves both approaches on the evaluation metrics but produces longer and more complex reports, and they note that agentic approaches may still be attractive for concise, orchestrated outputs. The paper frames the result as a reflection on when to use agentic frameworks in real-world applications.","pith_inferences":["Editorial inference: because DecomposedIR's reports are roughly three times longer (77.7 vs 25.0 sentences per report) and DecompEval aggregates sentence-level adherence, the 27% advantage could shrink substantially under length-controlled evaluation or independent human ratings.","Editorial inference: the same template-to-subquery recipe should transfer to other structured reporting tasks, such as regulatory filings, clinical notes, or equipment maintenance logs, where unstructured sources must fill a fixed schema; the paper does not test these domains.","Editorial inference: the results suggest reserving multi-agent architectures for tasks that genuinely need tool use, negotiation, or backtracking, and using linear prompt chaining for fixed-schema extraction; that is a cost-conscious reading the authors touch on but do not fully develop."],"forward_implications":["DecomposedIR is the stronger default for templated report generation when coverage of every template section matters, since it beat AgenticIR in two domains with different evaluation setups.","Self-reflection improves both frameworks on the four financial characteristics, so the two techniques can be combined rather than treated as exclusive alternatives.","The four-characteristic reference-free evaluation recipe (DecompEval and G-Eval) gives practitioners a no-gold-label way to compare report generators in the financial domain.","AgenticIR's weakness is attributed to missing explicit structural guidance, so adding template-structure constraints to agent workflows is a plausible way to close the gap."],"supporting_citations":[{"why":"Supplies the decomposed prompting method that DecomposedIR's template-to-subquery chain is built on.","marker":"[14]"},{"why":"Provides the multi-agent conversation framework used to implement AgenticIR.","marker":"[32]"},{"why":"Defines DecompEval, the reference-free metric that scores each sentence's adherence to key characteristics.","marker":"[11]"},{"why":"Defines G-Eval, the LLM rubric scorer used alongside DecompEval on the financial dataset.","marker":"[22]"},{"why":"Supplies the SumIPCC dataset with expert-written reference reports used for the cross-domain evaluation.","marker":"[6]"},{"why":"Provides the four key characteristics and the task framing that the financial evaluation is modelled on.","marker":"[7]"},{"why":"Supplies ROUGE, used to compare generated reports against expert-written SumIPCC references.","marker":"[20]"},{"why":"Supplies BERTScore, the embedding-based overlap metric used for reference comparison.","marker":"[36]"}],"fun_headline_variants":["Decomposed IR outshines agentic teams for templated reports","Prompt chaining beats agentic IR for financial report templates","DecomposedIR yields broader financial reports than AgenticIR","Splitting templates into questions beats agent collaboration","For templated reports, decomposed IR outperforms agentic teams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that DecompEval and G-Eval measure real report quality rather than rewarding length, since DecomposedIR writes about 78 sentences per report versus 25 for AgenticIR and DecompEval adds up sentence-level hits, a confound the paper does not resolve.","fun_headline_variants_meta":{"raw":{"variants":["Decomposed IR outshines agentic teams for templated reports","Prompt chaining beats agentic IR for financial report templates","DecomposedIR yields broader financial reports than AgenticIR","Splitting templates into questions beats agent collaboration","For templated reports, decomposed IR outperforms agentic teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2631,"prompt_tokens":993,"completion_tokens":1638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1554}},"tokens_in":609,"tokens_out":1638,"duration_ms":10840,"temperature":1.0,"reasoning_tokens":1554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:52:54.478388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have financial analysts rate a matched sample of AgenticIR and DecomposedIR reports for the same earnings calls, or recompute DecompEval on length-matched subsamples of the reports; if human preference or length-matched scores do not reproduce the reported 27% advantage, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the decomposed prompting method that DecomposedIR's template-to-subquery chain is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines DecompEval, the reference-free metric that scores each sentence's adherence to key characteristics."},{"cited_title":"Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models","cited_arxiv_id":"2411.14272","evidence_quote":"Supplies the SumIPCC dataset with expert-written reference reports used for the cross-domain evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the four key characteristics and the task framing that the financial evaluation is modelled on."}],"review_version":1}