{"id":"fa9bd79e-394f-42d0-b8dd-440a5e9837a9","arxiv_id":"2412.17942","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Contrato360 2.0 answers contract-management queries by combining RAG, text-to-SQL, and agent orchestration, but the claimed improvement over prior methods is not rigorously established.","lead":"A team from Brazil's BNDES development bank built a question-and-answer system that lets contract managers ask questions in plain language and get answers drawn from both contract PDFs and a contracts database. The system uses retrieval-augmented generation, text-to-SQL, and agent orchestration to avoid retraining the language model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of significant improvement is unsupported: Section 5 has no baseline, no statistical test, and one indirect-question type scores 0/10 correct.","rationale":"The paper describes a plausible engineering system that combines established techniques, and the authors provide a useful qualitative discussion of their design choices (e.g., section-based chunking, metadata filtering, SQL safety validation). These are real contributions and the system may well be useful in practice. However, the central scientific claim — that the multi-agent orchestration and combination of techniques 'significantly improve' relevance and accuracy — rests entirely on the Section 5 evaluation. That evaluation has the exact weaknesses the reader identified: no baseline, two non-blinded raters, no statistical analysis, and a question type that scored zero. My independent reading of the full text finds no additional internal inconsistency that would change the verdict; the primary issue is evidentiary. A conditional acceptance requiring a controlled comparison, independent blinded evaluation, and release of data/code is the right outcome. I therefore agree with the reader's verdict and recommend no change.","tokens_in":11512,"tokens_out":2155,"duration_ms":21340,"concrete_test":"Run a controlled ablation comparison on the same 75 contracts and the same benchmark questions: (1) the full Contrato360 2.0 system; (2) Contrato360 1.0 (RAG + static text-to-SQL, no agents); (3) RAG-only; (4) text-to-SQL-only. Have at least three independent, blinded domain experts rate each answer as Correct / Incomplete / Incorrect. Apply McNemar's test or a bootstrap paired test comparing full system versus each baseline on accuracy and completeness. If the full system does not significantly outperform the baselines (e.g., p < 0.05 or a pre-specified effect size), the central claim of significant improvement fails. Also report per-category results, including the DLs question that currently scores 0/10.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that multi-agent orchestration and the combination of RAG, text-to-SQL, agents, and prompt engineering 'significantly improve the relevance and accuracy of the answers.' The evaluation in Section 5 does not support this. There is no comparison against Contrato360 1.0 or against single-technique ablations, so the specific contribution of the multi-agent orchestration is not isolated. Only two BNDES IT specialists, who are presumably not blinded to the system's design, rated the answers; no inter-rater agreement, error bars, or statistical tests are reported. The most telling result is in Table 2: the question 'How many DLs (Exemptions from Tenders) were contracted in yy?' received 0 'Correct' and 10 'Incomplete,' and the authors admit the 'Waiver of Bidding' concept was not well captured. This is a direct counterexample to the claim of improved accuracy for a whole class of indirect questions. Additionally, Table 1 row 2 shows 9 correct and 1 incomplete, contradicting the statement that direct questions produced relevant answers 'for all experiments.' Without a baseline and a statistical test, the 'significant improvement' claim is an unsupported assertion, not a finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Contrato360 2.0, a question-answering system for contract management that combines Retrieval-Augmented Generation over contract PDFs, a text-to-SQL agent over a contract management database, prompt engineering, and router/graph agents orchestrated by an LLM. The system uses gpt-4-turbo for answer generation and OpenAI embeddings, with no model retraining. The authors report a pilot evaluation with 75 contracts and two BNDES IT specialists, classifying answers as Correct or Incomplete for both direct questions (answered from PDFs) and indirect questions (answered from database data), and they claim that the multi-agent orchestration and combination of techniques significantly improve answer relevance and accuracy.","tokens_in":11705,"tokens_out":4441,"duration_ms":43704,"significance":"If properly validated, the system would provide a useful industrial demonstration of LLM-based agents for enterprise contract management, with practical time savings and integration of unstructured documents with structured databases. The paper's strengths are its concrete architecture, the use of metadata-augmented chunking to address the similarity-versus-relevance problem in standardized contracts, SQL-command safety validation, and a real-world pilot with a Brazilian public bank. However, the current evaluation does not support the abstract's strong causal claim: there is no baseline, no ablation, no statistical analysis, and the reported results include a clear failure on an indirect-question class. The contribution is therefore best regarded as an engineering report whose central claim requires substantially stronger evidence.","major_comments":[{"comment":"The central claim that the multi-agent orchestration and combination of techniques 'significantly improve the relevance and accuracy of the answers' is not supported by the reported evaluation. Section 5 contains no comparison against Contrato360 1.0, no ablation in which agents, RAG, text-to-SQL, or prompt engineering are individually removed, and no statistical test or confidence measure. The word 'significantly' is used as a rhetorical claim rather than a measured one; the authors should either add a controlled comparison or replace this claim with a qualified statement about observed performance in a pilot setting.","section":"Abstract and Section 5"},{"comment":"The evaluation's measurement unit is unclear and the rater setup is under-specified. Each row in Tables 1 and 2 shows counts that sum to 10, but the text states that only two BNDES IT specialists were involved; it is not explained whether each rater answered each question five times, whether questions were re-run across sessions, or how the aggregate counts were produced. The raters are also not described as blinded to the system design or to the intended answer, and no inter-rater agreement is reported. Without this information, the 10-item accuracy counts cannot be interpreted as a reliable evaluation.","section":"Section 5, Tables 1 and 2"},{"comment":"The worst reported result is a direct counterexample to the claimed improvement for indirect questions: 'How many DLs (Exemptions from Tenders) were contracted in yy?' received 0 Correct and 10 Incomplete, and the authors attribute this to the 'Waiver of Bidding' concept not being well captured. The paper says that an adjustment in queries and/or prompts could add this semantics, but no follow-up experiment is shown. Because indirect questions over database records are a stated core contribution, this failure needs to be analyzed and addressed, not deferred.","section":"Section 5, Table 2"},{"comment":"The statement that the system presents relevant answers for direct questions 'for all experiments' is contradicted by Table 1, row 2, which shows 9 Correct and 1 Incomplete for the question about contract subject. The authors should either correct this statement or explain why an 'Incomplete' answer still counts as 'relevant'. A reader cannot reconcile the summary sentence with the table as printed.","section":"Section 5, Table 1"}],"minor_comments":[{"comment":"There is a typo in the distance metric list: 'Manhatan' should be 'Manhattan'. Please also check the spelling of 'Euclidean' and ensure the metric names are set in a consistent style.","section":"Section 2.2"},{"comment":"There are several language errors, including 'understanting' for 'understanding' and the sentence fragment 'In A question on the contract domain will follow our workflow to find a relevant answer.' The paragraph should be rewritten for clarity.","section":"Section 3.4"},{"comment":"The sentence 'The 8 demonstrates the ability of Contrato360 in' should read 'Figure 8 demonstrates the ability of Contrato360 to...'. More generally, every figure should be explicitly referenced in the text at the point where it is discussed.","section":"Section 6"},{"comment":"The footnote states that a third category 'Incorrect' was not obtained, but an 'Incomplete' answer is not necessarily a fully relevant answer. The binary Correct/Incomplete scheme should be justified, and the paper should clarify whether 'Incomplete' answers still count as successes or as partial failures in the reported claims.","section":"Section 5"},{"comment":"The LangChain documentation URL is broken across lines and should be given as a single, clickable link. Several citations are also formatted inconsistently (e.g., '(Li et al., 2022)' and '(Yao et al., 2023)'); please harmonize the reference formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is an application-oriented paper rather than a methods contribution, which is acceptable for the venue if the claims are matched to the evidence. The current gap between the abstract's 'significantly improve' and the pilot evaluation is the main blocker. I would not reject on scope alone, but the authors need either to add a proper baseline/ablation with statistical analysis or to explicitly reframe the paper as a descriptive system report. The self-citation to the authors' prior Contrato360 work is legitimate as background, but it should not be used as implicit validation of the current system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Contrato360 2.0 is a real engineering effort, not a toy. The authors build a Q&A system that pulls from both contract PDFs and a structured CMS, using clause-based chunking with metadata filtering, a router agent, a text-to-SQL agent, and prompt engineering. The chunking contribution is the most interesting part: contracts have standardized sections, so chunking by clause and storing contract/OCS/clause metadata helps disambiguate across similar contracts. That is a genuine, practical insight, and the Portuguese public-sector context gives it concrete value.\n\nThe paper honestly reports its evaluation, and the authors admit the weak spot: one indirect question type scored 0/10 because 'Waiver of Bidding' was not captured. That honesty counts. But the abstract claims the combination 'significantly improve[s] the relevance and accuracy,' and the evidence in Section 5 does not support that. There is no baseline, no ablation, no statistical test, no error bars, and the two raters are from the authors' own organization and not blinded. Table 1 row 2 shows 9 Correct / 1 Incomplete for a direct question, which actually contradicts the sentence that direct questions produced relevant answers 'for all experiments.' Table 2's 0/10 on DLs is a direct counterexample to the claim of improved indirect-question accuracy.\n\nNone of this means the system is bad. It probably works reasonably well for the demonstrated query types. But the paper as written overclaims, and a reader cannot tell whether the multi-agent orchestration adds value over the authors' prior Contrato360 1.0 or over a simple RAG+SQL pipeline. The fix is straightforward: add a baseline comparison against 1.0 and against single-technique configurations, report per-question types with confidence intervals or at least raw counts, and consider a blinded or at least independent rater. Releasing code or data would also go a long way.\n\nWho gets value from this paper: practitioners building enterprise Q&A over contracts, especially in Portuguese; researchers studying chunking or agent orchestration will find the metadata-filtered clause chunking worth a look. The evaluation needs major work before publication, but the engineering and the honest account of failure cases make it worth a serious referee.","headline":"Useful applied system with a real chunking idea, but the headline improvement claim is not supported by the paper's own evaluation; deserves revision, not rejection.","tokens_in":12249,"tokens_out":2323,"would_cite":false,"duration_ms":21787,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Router agent answers contract questions from PDFs and databases","keywords":["question answering","contract management","retrieval-augmented generation","text-to-SQL","LLM agents","prompt engineering","vector databases","enterprise information systems"],"falsifier":"Run the same direct and indirect benchmark questions against Contrato360 1.0, against a RAG-only configuration, and against a SQL-only configuration, with raters blind to which version produced each answer. If the full agent system does not achieve a higher correct rate than the baselines on the same question set, the claim of significant improvement fails. A simpler check: recruit raters who did not help write the questions and see whether the indirect-question scores, especially the waiver-of-bidding and exemption-from-tender items, stay at the reported level.","tokens_in":11317,"feed_emoji":"🤖","tokens_out":7030,"duration_ms":62114,"temperature":0.7,"pith_summary":"The paper presents Contrato360 2.0, a question-answering system for contract management that draws on two sources at once: the text of contract PDFs and the structured records of a contract management database. The authors claim that routing every question through a router agent to a retrieval-augmented generation (RAG) agent and a text-to-SQL agent, then assembling the answer with prompt-engineered instructions, makes answers more relevant and accurate than using any single technique on its own, and does so without retraining the language model. They report that on a benchmark of prepared questions, specialists rated 59 of 60 direct questions correct and 65 of 80 indirect questions correct, with the remainder incomplete. A sympathetic reader would take the paper's contribution to be an architecture showing that enterprise document-and-database Q&A can be assembled from orchestrated components rather than fine-tuned models.","feed_headline":"Router agent answers contract questions from PDFs and databases","feed_subtitle":"RAG, text-to-SQL, and prompt engineering answer contract queries from documents and databases with no model retraining.","key_machinery":"The load-bearing mechanism is the multi-agent orchestration workflow, in particular the router agent that acts as the decision-maker for every query. In parallel, a RAG agent retrieves document chunks from a vector store and a SQL agent retrieves structured records from the contract database; the retrieved pieces are then combined by prompt engineering into the final answer. A supporting mechanism is metadata-enriched chunking: contracts are split by clause rather than by fixed token count, and each chunk carries the source file, the OCS contract number, and the clause title. The metadata filter is what prevents querying one contract from returning a textually similar clause of a different contract, addressing the paper's stated similarity-versus-relevance problem in standardized legal documents. The SQL agent is constrained to read-only queries, which the authors present as the safety design of the system.","core_discovery":"On its own terms, the paper's claim is that a question-answering system can be made substantially more useful for contract managers by replacing a single static pipeline with four coordinated techniques: section-based chunking of contract PDFs enriched with metadata (source file, contract number, clause), retrieval-augmented generation over the resulting vector store, a text-to-SQL agent that maps natural-language questions onto the contract database schema, and prompt engineering that fixes response style and forces answers to cite the contract number. The router agent decides whether a question belongs to the contract domain and then sends in-domain questions to the RAG and SQL agents in parallel; when the retrieved material includes tabular values, a graph agent can add a visualization. The authors' reported evaluation is a benchmark of prepared direct and indirect questions judged by two domain specialists, and their conclusion is that the agent orchestration combines both data sources well enough to answer most questions correctly without any fine-tuning of the language model. The paper frames this as a general direction for information systems: compose RAG, text-to-SQL, agents, and prompt engineering rather than retraining a model for each new document corpus.","pith_inferences":["The paper's accuracy conclusion is not yet supported by a comparison: the benchmark lacks a run against the earlier Contrato360 version or against configurations with individual techniques removed, so the reader cannot tell which component contributes the gain.","The metadata-filtering fix suggests a direct ablation test: removing the contract-number metadata from chunks should measurably lower retrieval precision on cross-contract questions such as 'what is the object of contract OCS 278/2023'.","The router's parallel split between RAG and SQL paths raises a boundary question: for a query that needs both a database figure and a document clause, the system retrieves both in parallel and merges them in the prompt, so the merge step is where a wrong answer would hide.","The architecture generalizes to any regulated document-heavy workflow where the same entity appears in both free-text documents and structured records; the cost is engineering per-domain metadata and schema, not model training."],"forward_implications":["Contract managers can ask questions that span a PDF and the database, such as the contract manager plus the penalty clause, and receive one consolidated answer instead of searching documents by hand.","New contracts can be added to the system by chunking and embedding their PDFs and loading their records into the database; the language model itself does not need to be retrained for the new content.","Because text-to-SQL handles the structured half of the query, users do not need to know SQL or the database schema to get exact figures from the contract system.","The router's domain filter gives the system a bounded scope, so open-domain questions are deflected rather than answered from the model's general knowledge.","The same composition of RAG, text-to-SQL, agents, and prompt engineering is presented as transferable to other document-plus-database enterprise domains."],"supporting_citations":[{"why":"The earlier Contrato360 system that used RAG and a static text-to-SQL approach; this paper's agent orchestration extends it.","marker":"(Seabra et al., 2024)"},{"why":"Supplies the SQL agent implementation used to connect to the contract database and generate queries.","marker":"(Langchain, 2024)"},{"why":"Motivates RAG as a remedy for hallucination, outdated knowledge, and lack of domain expertise in LLMs.","marker":"(Chen et al., 2024)"},{"why":"Provides the RAG survey and retrieval-pipeline framing that the vector-store approach follows.","marker":"(Gao et al., 2023b)"},{"why":"Establishes augmented language models and agent-based orchestration as the basis for routing queries to specialized modules.","marker":"(Mialon et al., 2023)"},{"why":"Foundational retrieval-augmented generation formulation that the paper's RAG component relies on.","marker":"(Lewis et al., 2020)"},{"why":"Supports using LLM reasoning-and-acting for the text-to-SQL decision task in the system.","marker":"(Yao et al., 2023)"}],"fun_headline_variants":["Contract Q&A bot fuses PDFs and SQL via agents","No retraining: router agent answers from contracts and DB","Multi-agent system answers contract queries from docs and data","RAG plus text-to-SQL agents field contract questions","Agents combine PDFs and databases for contract Q&A"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of significantly improved accuracy rests on a benchmark in which two specialists from the deploying organization rated the system's answers to questions the authors prepared, with no comparison against the earlier version of the system or against a configuration missing one of the four techniques; if those ratings are not representative, the central claim is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Contract Q&A bot fuses PDFs and SQL via agents","No retraining: router agent answers from contracts and DB","Multi-agent system answers contract queries from docs and data","RAG plus text-to-SQL agents field contract questions","Agents combine PDFs and databases for contract Q&A"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1313,"prompt_tokens":911,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":321}},"tokens_in":527,"tokens_out":402,"duration_ms":4265,"temperature":1.0,"reasoning_tokens":321,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:07:48.056192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same direct and indirect benchmark questions against Contrato360 1.0, against a RAG-only configuration, and against a SQL-only configuration, with raters blind to which version produced each answer. If the full agent system does not achieve a higher correct rate than the baselines on the same question set, the claim of significant improvement fails. A simpler check: recruit raters who did not help write the questions and see whether the indirect-question scores, especially the waiver-of-bidding and exemption-from-tender items, stay at the reported level.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier Contrato360 system that used RAG and a static text-to-SQL approach; this paper's agent orchestration extends it."},{"cited_title":"Langchain agents documentation","cited_arxiv_id":null,"evidence_quote":"Supplies the SQL agent implementation used to connect to the contract database and generate queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates RAG as a remedy for hallucination, outdated knowledge, and lack of domain expertise in LLMs."}],"review_version":1}