{"id":"b7d60101-5e3e-4f48-a3e6-a72c293f4d5f","arxiv_id":"2412.09280","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A pipeline that generates executable programs from domain knowledge documents and uses them with extracted variables to solve domain-specific calculation problems, improving accuracy over baselines in legal and medical QA.","lead":"KIPG is a pipeline that turns domain rule documents (such as Chinese law articles) into executable Python programs, then uses those programs to answer numeric questions. In legal and medical benchmark tests it beats standard prompting and program-of-thought baselines, and the code generator generalizes to new domains without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DPO program selection uses si computed from training queries (Sec. 4.3); programs can score 1.0 by covering only the sampled branch of a piecewise rule and still fail on unseen queries, so KIPG's reported 8.77% relative gain may be selection overfit.","rationale":"The reader's weakest assumption correctly identifies the validity of si as the load-bearing premise, and I agree with that identification. The paper's central empirical claim, that knowledge-intensive programs significantly improve performance over all baselines, rests almost entirely on the DPO-trained code generator: without training, KIPG falls to 60.93 average accuracy, below EEDP at 64.23 (Table 7). The ranking signal that drives the improvement is si, computed by executing each candidate program on the training queries for its article and comparing to gold answers. For this to support the claim, si must be a reliable estimate of the program's correctness on unseen queries of the same article. The legal and medical rules in the dataset are naturally piecewise, with different queries activating different condition branches; Figure 2 is an explicit illustration of this phenomenon. A program that happens to implement only the branches represented in the training queries can receive perfect si while being systematically wrong on other branches. Appendix C provides only an aggregate correlation between training-subset and test-set program accuracies, which does not establish branch coverage for the specific top-ranked program. Thus the reported 8.77% relative improvement may be an artifact of training-query distribution rather than a general ability to follow knowledge-intensive documents. The proposed branch-held-out test would settle this directly: if selected programs retain accuracy on unseen branches, the concern is refuted; if not, the central claim needs to be qualified. I do not see this as a reason to change the reader's CONDITIONAL verdict, because the concern is concrete but currently unverified, and the requested artifacts and branch-level analysis can be added without altering the paper's overall structure.","tokens_in":16566,"tokens_out":7547,"duration_ms":86726,"concrete_test":"For each article in the released (or to-be-released) dataset, identify the condition variables that define disjoint cases (e.g., property-amount brackets, fixed-income status). Split each article's queries into a training-branch subset T and a held-out-branch subset H so that at least one operative branch is absent from T. Recompute si using only T, select the top-ranked program, and measure KIPG accuracy on H; compare this with accuracy of the same program when si is computed on all queries. If branch-held-out accuracy is close to full-data accuracy for all articles, the proxy holds. If any top-ranked program drops by more than 5 points on H, DPO selection is overfitting to the training branch. A simpler variant is to build 20-30 adversarial queries whose values fall in brackets not present in the current training instances and compare KIPG against EEDP only on those queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing premise is that the correctness score si used to build DPO preferences in Sec. 4.3 is a valid proxy for how the same program will behave on unseen queries of the same article. The pipeline deploys one program per document: programs are sorted by si, trained with DPO, and then used offline to compute (I,O) for every test query. Legal and medical rules are strongly piecewise (amount brackets, fixed vs non-fixed income, maximum vs minimum cases), and Figure 2 explicitly shows different queries exercising different branches of the same article. Nothing in the construction of si forces branch coverage: if the training queries for an article happen to fall in one bracket, a program that hard-codes that bracket gets si near 1.0 even though it would fail on any query outside the bracket. The paper's only indirect check, Appendix C / Figure 7, plots aggregate program accuracy on a 330-query subset against a 2050-query set; it does not test whether the particular program selected by si covers branches absent from the training queries. The stakes are high because Table 7 shows the w/o-training variant (60.93) is below the best baseline EEDP (64.23), so essentially all of KIPG's headline gain is produced by DPO selection. If si is not branch-general, the central claim is an artifact of training-query coverage rather than a general ability to follow knowledge documents. For the unseen types Penalties and Traffic Violations, the paper does not specify how correctness scores are computed at all, yet Table 2 reports strong gains there, which makes the proxy question even more acute for the 'all types' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces KIPG, a pipeline for domain-specific calculation QA in which a generator LLM writes \\u201cknowledge-intensive programs\\u201d from domain documents, an extraction model maps each query to program inputs, an executor computes the outputs, and a conclusion LLM turns the extracted variables and program outputs into a final answer. The generator is trained with iterative Direct Preference Optimization (DPO), where the preference signal is an execution-based correctness score s_i computed by running each candidate program on a small set of queries. The authors construct a Chinese legal dataset spanning five case types plus a medical cross-domain test set, and they report experiments with oracle and retrieved documents, cross-domain generalization, an English/Llama-3 setting, and ablations. The central claim is that KIPG outperforms the baselines, with an average accuracy of 69.86% on the legal oracle-context benchmark using a 7B conclusion model, quoted as 8.77% relatively above the best baseline EEDP.","tokens_in":16915,"tokens_out":9240,"duration_ms":90462,"significance":"If the claims hold, KIPG is a useful contribution: it proposes a program-generation paradigm that captures the conditional, branch-structured logic of domain documents rather than query-specific or terminology-only programs, and it uses execution against gold answers as an external correctness signal for DPO. The paper is also strong in the breadth of its evaluation: oracle and no-oracle settings, unseen case types, cross-domain transfer to medical data, cross-language transfer, multiple base LLMs, and ablations. The Limitations section honestly acknowledges that retrieval is not optimized and that accuracy drops by more than 10% without the label document. However, the headline results rest on two load-bearing assumptions that are not yet demonstrated: that the s_i score used to select programs generalizes across the branches of piecewise rules, and that the evaluation is conducted on a clearly disjoint held-out set. These issues must be resolved before the empirical claims can be considered robust.","major_comments":[{"comment":"The correctness score s_i that ranks programs for DPO and for final program selection is computed on a small \\u201ctraining set\\u201d of 330 queries (Appendix C). Figure 7 only compares aggregate program accuracy on that set with the 2050-query set; it does not test branch-level generalization. Because many legal articles are piecewise (income brackets, fixed vs. non-fixed income, maximum vs. minimum cases), a program that hard-codes the branches represented in the 330 queries can receive a high s_i and still fail on unseen branches. This concern is load-bearing because Table 7 shows that removing DPO training lowers the average accuracy by 6.73 points and by 11.94 points on Other Fees, so most of the headline gain is produced by the s_i-based selection. Please add a per-article analysis of whether the selected program generalizes to queries that exercise branches absent from the selection set, or select programs on a held-out set and report accuracy on those held-out branches.","section":"Sec. 4.3, Appendix C, Table 7"},{"comment":"The paper never specifies whether the 330-query program-filtering set overlaps with the 2050-query evaluation set. The total number of legal instances in Table 1 is exactly 2050, so the \\u201ctesting set\\u201d in Figure 7 appears to be the entire dataset. If the 330 queries used to compute s_i are part of the evaluation set, the reported accuracies are not fully held out. Similarly, for the unseen types Penalties and Traffic Violations, Section 5.1 says their calculation instances are never seen during training, but computing s_i on queries for those types would make those queries seen during program selection. Please state the exact query-level split, ensure disjointness between the selection set and the evaluation set, and re-report any affected numbers.","section":"Appendix C, Table 1, Sec. 5.1"},{"comment":"All results are single runs without standard deviations, confidence intervals, or significance tests. Several comparisons are close: for example, in Table 2 the Qwen2-72B Compensation result for PoT is 57.27 versus 56.82 for KIPG, and in Table 4 the Indicator column shows EEDP at 48.53 versus KIPG at 52.94. Without repeated runs or a paired test, these margins may not be reliable. Please provide at least three seeds per condition, or bootstrap confidence intervals, and a paired significance test where the comparison is between methods on the same queries.","section":"Tables 2\\u20137"},{"comment":"The dataset and code are not released, and the appendix provides no link, no detailed annotation protocol beyond a summary, and no public access to the generated programs. Since the central evidence is empirical and the dataset is newly constructed, the inability to reproduce or inspect the data blocks independent verification of the branch-coverage and split concerns above. Please release the dataset, the code, and the sampled programs, or provide a clear statement of why they cannot be released.","section":"Data availability"}],"minor_comments":[{"comment":"The text says CodeQwen\\u2019s average accuracy is 60.43%, but Table 5 reports 60.45%; please harmonize the number.","section":"Sec. 5.4, Table 5"},{"comment":"The sentence \\u201cthe knowledge-intensive programs significantly improve the overall performance under all types of cases\\u201d should be qualified for the 72B setting, where Table 2 shows PoT is higher on Compensation (57.27 vs. 56.82).","section":"Sec. 5.1, Table 2"},{"comment":"The conditioning variable d is used in the DPO loss but is not defined in the main text; from context it is the domain document or article, but please define it explicitly.","section":"Eq. (3)"},{"comment":"The two dialogue panels are difficult to parse in the current rendering; the caption should state what each panel contains and how the two responses differ in their calculation logic.","section":"Figure 2"},{"comment":"The ablation in Table 7 is reported for the second iteration, but Table 2 does not state which iteration the main results use; please clarify whether the main tables correspond to the final iteration or a specific iteration.","section":"Sec. 5.6, Table 7"},{"comment":"The sentence \\u201cIt can be seen that there is a high consistency between the two accuracy\\u201d is purely visual; please quantify the agreement or correlation between the small-set and large-set program accuracies.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially suitable for the journal if the branch-coverage and data-split issues are resolved. The manuscript\\u2019s own Limitations section candidly acknowledges the retrieval gap, but it does not acknowledge the more serious risk that s_i-based program selection overfits to the 330-query filter set. I would ask the authors, at minimum, for the exact query-level split, confirmation that the selection set and evaluation set are disjoint, and a branch-coverage analysis showing that the selected programs handle branches absent from the selection set. The lack of released data and code further hampers verification of these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the core idea is genuinely new: instead of generating a program per query (PoT) or per terminology (SciAgent), KIPG writes one program per knowledge document that encodes the full rule logic and returns all potential outcomes, then uses a 7B model to extract variables, execute, and let a conclusion model pick the answer. Second, the experiments are unusually thorough for this kind of paper—oracle and no-oracle settings, cross-domain, cross-language, ablations over every module, and an honest limitations paragraph. The reported gains are consistent and often large: 69.86% vs 64.23% for the best baseline on the 7B oracle benchmark, and it beats stronger backbones on medical and English-translated data. I believe the central claim, that KIPG improves accuracy by encoding documents as programs and training the generator with execution-based DPO, is supported by the reported tables.\n\nThe soft spots are real but not fatal. The most load-bearing concern is that the DPO preference score si is computed by running each candidate program on the training queries for its article. Legal and medical rules are piecewise, and Figure 2 shows that different queries exercise different branches. Nothing forces branch coverage: a program that hard-codes the one bracket present in the training queries can get si = 1.0 and still fail on any query outside that bracket. The only indirect check, Appendix C's aggregate train/test accuracy plot, does not test per-branch generalization, and the paper does not explain how programs for the held-out types (Penalties, Traffic Violations) are scored at all, even though Table 2 reports strong gains there. This is exactly the place where a reviewer should dig. A second, more mundane issue: no code, no data, no error bars, and all numbers are single runs, so the magnitude of the gains is unverifiable from the paper alone. That is a common preprint weakness but it matters more here because the whole pipeline is a concrete engineering contribution that others would want to reproduce.\n\nWho is this for? Anyone working on LLM reasoning with domain documents, tool-augmented QA, or preference optimization with executable feedback. It deserves a serious referee—the idea is clean, the evaluation is broad, and the branch-coverage question can be settled with a targeted analysis. I would send it to review with a request for artifacts and a per-branch program-accuracy study. I would not desk-reject it.","headline":"Solid method paper with a real empirical claim; the main risk is that DPO program selection may overfit to training-query branches, and the missing artifacts need to be addressed.","tokens_in":17434,"tokens_out":1472,"would_cite":true,"duration_ms":17623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-specific calculation problems are best solved by generating knowledge-intensive programs that encode each document's full rule logic, and a code generator trained by execution-based preference optimization can write those programs…","keywords":["knowledge-intensive programs","domain-specific calculation","code generation","direct preference optimization","legal QA","numerical reasoning","program-of-thought","cross-domain generalization"],"falsifier":"To test the proxy, hold out every query that exercises one clause of an article, train KIPG only on queries that avoid that clause, then measure accuracy on held-out clause queries; if accuracy on those queries is no better than a generator trained without the correctness scores, the assumption fails.","tokens_in":16395,"feed_emoji":"⚖️","tokens_out":7272,"duration_ms":67544,"temperature":0.7,"pith_summary":"Domain-specific calculation problems — for example, computing litigation fees or lost wages from a law article — are hard for LLMs because the governing knowledge is a set of conditional rules rather than a simple formula. This paper claims that the right unit of computation is a knowledge-intensive program: executable code written once per document that encodes all of the document's conditions and returns every outcome the document allows. On top of that, the paper trains a code generator with iterative direct preference optimization, ranking candidate programs by how often they compute the correct answers on training queries for the same article. With a 7B conclusion model, the resulting pipeline reaches 69.86% average accuracy on legal oracle-document cases, about 8.8% relatively above the best prompting baseline, and the same generator transfers to unseen case types and to the medical domain.","feed_headline":"Legal rules as executable programs lift QA to ~70%","feed_subtitle":"A 7B model writes per-article code that computes every allowed outcome, beating much larger models on numeric legal questions.","key_machinery":"The load-bearing object is the knowledge-intensive program: a Python function generated for each article or document, with a header comment naming the knowledge source, typed input/output parameter descriptions, and inline citations of the original rule before each branch. Unlike query-oriented programs that solve one question or terminology-oriented programs that return one concept's value, it returns all outcomes permitted by the document, so the same program serves every query about that article and query phrasing cannot bias the calculation. The training machinery is iterative DPO: candidate programs are sampled with diverse beam search, executed on training queries for their article, scored by matching the gold numeric answers, and ranked; the resulting preference pairs update the generator, and the loop repeats.","core_discovery":"The central claim is that replacing query-specific or terminology-level code with article-level knowledge-intensive programs, and teaching a generator to write those programs by execution-based preference alignment, makes LLMs reliably follow complex domain rules in numeric QA. A knowledge-intensive program is generated once for each domain document; it contains hierarchical if/else conditions, comments citing the source sentences, and returns a dictionary of all potential outcomes rather than one answer. At inference, KIPG extracts variables from the query, executes the program, and prompts a conclusion model to derive the final answer. The paper reports that this decomposition yields the highest accuracy in both oracle-context and retrieval settings, that correctness of programs on training queries tracks correctness on a larger held-out set closely enough to serve as a training signal, and that a generator trained only on three legal case types still improves accuracy on unseen legal case types and on medical documents.","pith_inferences":["Beyond this paper, the same loop — generate executable artifacts, score them by execution on checkable queries, rank with DPO — could apply wherever outputs are verifiable, such as spreadsheet formulas, data-cleaning pipelines, or rule-based policy engines.","Because the paper reports a more than 10% average accuracy drop when the gold document is not given, the program generator's full value is currently gated by retrieval quality; coupling KIPG with a retriever trained on rule-condition logic rather than surface embeddings is a testable next step.","The authors' observation that adding documents without calculation examples still improves correctness suggests the generator first learns a general 'rules as code' syntax; an experiment measuring program accuracy on those syntax-only articles would separate that effect from answer-matching.","The English Llama3 results show baseline rankings shift across languages, with Program of Thought far stronger there, so the transfer benefits of knowledge-intensive programs may depend on the base model's language distribution rather than on the method alone."],"forward_implications":["Given the gold document, KIPG reaches 69.86% average accuracy with a 7B conclusion model, which is 8.77% relatively above the best prompting baseline and at least matches a much larger 200B API model on several case types.","Outputting all possible outcomes instead of a query-specific scalar removes query-induced calculation bias, so one program per article can be reused across an arbitrary number of questions.","The code generator's skill transfers to legal case types never seen in training, and to medical-domain documents, without retraining on the new knowledge.","Supplying the calculated variables from the smaller generator to a 72B conclusion model raises accuracy from 69.86% to 74.84%, so the benefit of knowledge-intensive programs is not limited to small models.","Iterating DPO generally raises accuracy over rounds, and the correctness of programs on a small training set closely tracks correctness on a larger test set, so automatic filtering can keep inference cheap."],"supporting_citations":[{"why":"Supplies the DPO objective used to align the program generator with preference pairs built from execution scores.","marker":"(Rafailov et al., 2024)"},{"why":"Defines Program of Thought, the query-oriented program baseline that KIPG is designed to improve on.","marker":"(Chen et al., 2023)"},{"why":"Introduces SciAgent and terminology-oriented programs, the closest prior use of external-knowledge programs that KIPG contrasts.","marker":"(Ma et al., 2024)"},{"why":"Supplies the strongest baseline comparison in the oracle-context experiments, against which KIPG's relative gain is stated.","marker":"(Srivastava et al., 2024)"},{"why":"Diverse beam search is the sampling mechanism that produces the candidate program set for DPO data.","marker":"(Vijayakumar et al., 2018)"},{"why":"Frames the task of knowledge-intensive math reasoning in a specific domain and the query-oriented approach.","marker":"(Zhao et al., 2024)"},{"why":"GSM8K serves as the general math training mix used in the supervised fine-tuning baseline.","marker":"(Cobbe et al., 2021)"},{"why":"Qwen2-7B and 72B are the base LLMs whose instruction-tuned versions serve as generator, extractor, and conclusion model.","marker":"(Yang et al., 2024)"}],"fun_headline_variants":["KIPG codes up legal rules to ace numeric QA","7B model writes article-programs, lifts legal QA to ~70%","Knowledge-intensive programs: documents to code for complex math","Generated programs from docs help LLMs do domain-specific calculations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking signal in DPO training is a program's correctness on the training queries for its article, and that score must be a reliable proxy for correctness on unseen queries of the same article; a program that fits training cases while misencoding a rarely tested rule would survive training and fail at test time.","fun_headline_variants_meta":{"raw":{"variants":["KIPG codes up legal rules to ace numeric QA","7B model writes article-programs, lifts legal QA to ~70%","Knowledge-intensive programs: documents to code for complex math","Generated programs from docs help LLMs do domain-specific calculations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3300,"prompt_tokens":901,"completion_tokens":2399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2329}},"tokens_in":517,"tokens_out":2399,"duration_ms":18417,"temperature":1.0,"reasoning_tokens":2329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:37.217351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test the proxy, hold out every query that exercises one clause of an article, train KIPG only on queries that avoid that clause, then measure accuracy on held-out clause queries; if accuracy on those queries is no better than a generator trained without the correctness scores, the assumption fails.","supporting_citations":[],"review_version":1}