{"id":"22537b07-194e-47af-b663-9894bf96c3e6","arxiv_id":"2411.16454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Retrieving few-shot examples by computational-graph similarity improves LLM math word problem accuracy by up to 6.7 points over semantic retrieval, without retraining the generator.","lead":"This paper trains a retriever to find math word problems that share the same underlying calculation structure, then uses those problems as examples when prompting an LLM. Across six datasets, graph-based example selection beats random and semantic-similarity retrieval, with the largest gains on smaller models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retriever's graph-structure alignment is only validated on Math23k; on the five other datasets the mechanism behind the accuracy gains is not directly tested, so domain adaptation remains a plausible alternative explanation.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the retriever trained on Math23k must transfer structural similarity to five other datasets, a condition on which the paper provides only indirect evidence. My analysis agrees. The paper does not include error bars or significance tests, and the correlation analysis in Figure 4 is in-distribution, so the central mechanism is not directly validated out-of-distribution. However, the empirical results are broadly positive, and the method's behavior on math_qa (where gains vanish) is consistent with the proposed mechanism, so the evidence is not sufficient to reject the claim. The natural next step is a targeted test on Calc-ape210k, where equation solutions permit graph similarity measurement. Since the reader already issued a conditional accept pending additional validation, my stress-test does not change that verdict; it reinforces it with a concrete test. A potential alternative concern—the lack of statistical significance—is real but secondary, and the prescribed test would also partially address it by quantifying retrieval quality directly. Therefore, the verdict remains CONDITIONAL, and no adjustment is needed.","tokens_in":935,"tokens_out":848,"duration_ms":57904,"concrete_test":"On Calc-ape210k, which has equation-form solutions, sample 200 test queries and, for each, retrieve top-8 exemplars from its training set using both the trained retriever and the BGE baseline. Parse each query's and each retrieved example's equation into a normalized computational graph string (canonicalizing operand order and variable names), then compute graph similarity (exact match or normalized Levenshtein distance) between query and each retrieved exemplar. Compare the mean graph similarity of the retrieved sets. If the trained retriever does not yield significantly higher graph similarity than BGE, the mechanism is not transferring to unseen datasets, and the accuracy gains are better attributed to domain adaptation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that computational-graph-based retrieval improves few-shot MWP accuracy by selecting exemplars whose hidden arithmetic structure matches the query (Section 1, Section 2). The retriever is trained on 25% of Math23k (Section 3.1), using positive pairs defined by exact equality of the provided computational graph templates (G+_i = G_i, Section 2.2). The only direct evidence that the trained embeddings actually align with graph structure is the scatter-plot correlation in Figure 4, but that analysis is computed on Math23k itself—the training distribution. For the other five datasets, no quantitative measure of graph similarity between queries and retrieved exemplars is reported; the only support is a small qualitative case study on Calc-ape210k (Section 3.3.2, Figure 3). Consequently, the observed EM gains over BGE (Table 2) could be explained by the retriever being fine-tuned on math-domain text, improving retrieval of semantically relevant examples, without necessarily retrieving structurally analogous ones. If the active ingredient is domain adaptation rather than graph analogy, the paper's central contribution—and its claimed advantage over semantic retrieval—is not established for out-of-distribution data. This is load-bearing because the headline average gain of up to 6.7 points depends on datasets (Math23k, ape210k, Calc-ape210k) where equation templates are available and likely correlate with surface form; on gsm8k, math_qa, and aqua_rat the improvements are small or absent, which is consistent with the transfer concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-augmented few-shot prompting method for math word problems (MWPs). The retriever is an encoder trained with a contrastive objective on Math23k question pairs whose computational graph templates are identical; at inference it retrieves the top-k most similar question-solution pairs from the training corpus and prepends them to the query as few-shot exemplars for an LLM generator. The generator is not retrained. The authors evaluate on six MWP datasets (Math23k, ape210k, gsm8k, math_qa, Calc-ape210k, aqua_rat) with generators from 0.5B to 70B parameters, reporting exact-match accuracy. They report average gains of up to 6.7 points over BGE retrieval and 19.5 points over random selection, plus analyses of an upper bound on Math23k, a qualitative case study, a correlation scatter plot, training-data scaling, corpus mismatch, and LLM-distilled training data without human labels.","tokens_in":12666,"tokens_out":6926,"duration_ms":65876,"significance":"If the empirical gains are reliable, the method is practically attractive: it is modular, requires no generator fine-tuning, and the analysis in Section 3.3.3 suggests that only a few thousand labeled pairs suffice. The paper also makes a good-faith effort to study the mechanism, including a direct upper-bound comparison on Math23k and a corpus-shift experiment, and it is transparent about the limitations of the distilled-data variant. The main weakness is evidentiary: all accuracy numbers are single runs, and the central claim that the retriever selects structurally analogous exemplars is directly validated only on the retriever's own training distribution. The current evidence supports the conclusion that fine-tuning the retriever on math text improves few-shot prompting, but it does not yet establish that computational-graph structure, rather than improved semantic/domain matching, is the active ingredient for the out-of-distribution datasets.","major_comments":[{"comment":"All results in Table 2 are single runs; no seeds, standard deviations, confidence intervals, or significance tests are reported. Several per-dataset differences are small or negative: gsm8k with LLaMA-3B is +0.4 over BGE, and math_qa is -0.7, -0.6, and -0.5 for the 0.5B, 3B, and 70B generators, respectively. Without variance estimates, the abstract's claim of 'significant improvement' and the headline 6.7-point average cannot be properly assessed; the average also hides multiple datasets where the proposed method does not improve over BGE.","section":"Section 3.2, Table 2"},{"comment":"The central mechanism claim is that the trained retriever selects exemplars whose computational graphs match the query's graph. This is directly quantified only on Math23k (Figure 4), which is the retriever's training distribution. For ape210k, gsm8k, math_qa, Calc-ape210k, and aqua_rat, no quantitative measure of graph similarity between queries and retrieved exemplars is reported; the Calc-ape210k case study (Figure 3) is qualitative. Since the retriever is fine-tuned on math text, the observed gains in Table 2 could arise from improved semantic or domain-specific retrieval rather than from computational-graph analogy. This is load-bearing because the claimed advantage over BGE rests on structural alignment. Please report graph-template overlap or normalized graph-string similarity between query and retrieved exemplars on datasets with equation solutions (ape210k, Calc-ape210k), or include a control retriever fine-tuned on math-domain contrastive pairs without graph labels to show that the gains require the graph-based positive pairs.","section":"Section 3.3.2, Figure 4, and Table 2"},{"comment":"The authors state that in the distilled-data training, 'the sentence structure before and after rewriting is more similar than in the labeled data pairs, which the retriever may rely on to capture similarity between positive pairs during training, rather than focusing on the true computational graphs.' This is a self-admitted limitation of the Section 4 pipeline, and it directly undermines the claim that the distilled-data retriever works by graph analogy. It also reinforces the concern in the previous comment: the retriever can improve downstream accuracy without necessarily encoding graph structure. The Section 4 claim of 'computational graph-free' training data should be qualified, and the sentence-structure confound should be measured, for example by comparing lexical or syntactic overlap between positive pairs in the labeled and distilled settings.","section":"Section 4"},{"comment":"The label 'upper bound' for retrieval by normalized Levenshtein distance on graph strings is not strictly justified. It assumes that graph-similar retrieval is the optimal retrieval policy for the generator, but the generator could in principle benefit more from some other selection criterion. The comparison is still informative as a sanity check on retriever quality, but the terminology overclaims and should be softened, for example to 'graph-oracle reference' or 'graph-retrieval reference'.","section":"Section 3.3.1, Table 3"}],"minor_comments":[{"comment":"The column header 'A vg.' appears to be a typo for 'Avg.'.","section":"Table 2"},{"comment":"The caption in Figure 2 misspells 'Retriver' as 'Retriver'; in Figure 4 the p-values are rendered as '4.97e 54' and '8.87e 05', which should be '4.97e-54' and '8.87e-05'.","section":"Figure 2 and Figure 4"},{"comment":"The text mentions 'Claude-3.5 or GPT-4' as the LLM used for data distillation, while the Table 5 caption says 'distilled from GPT-4o'; please make the model reference consistent.","section":"Section 4 and Table 5"},{"comment":"There are typos: 'availale' appears in the text and 'not availale' in the Table 4 caption; also, the phrase 'the training set of SuperCLUE-Math6 is not availale' should be clarified as 'not available'.","section":"Section 3.3.4"},{"comment":"The citation 'Toshniwal et al.;' in the Related Work section lacks a year and has a stray semicolon; it should be formatted consistently with the other references.","section":"Related Work"},{"comment":"The description of Figure 4 as 'the top-8 retrieved data points from 100 random samples' is ambiguous: please clarify whether this means 100 random queries with 8 retrieved items each, and how the scatter plot was aggregated.","section":"Section 3.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely topic. I would not recommend rejection: the central empirical finding, that fine-tuning the retriever on graph-positive pairs improves few-shot MWP accuracy, is plausible and potentially useful. The main blocker is that the causal mechanism is not established for out-of-distribution datasets, and the statistical support is missing. If the authors cannot provide OOD graph-similarity analysis or a suitable control retriever, they should substantially soften the causal claims and present the method as a domain-adapted retrieval technique rather than as confirmed evidence for computational-graph analogy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: solid, useful empirical paper. Training a contrastive retriever on computational-graph equality to pick few-shot exemplars for math word problems is a new combination, and it works consistently across six datasets and five generator sizes. It deserves a serious referee, though the mechanism evidence has a gap on out-of-distribution transfer.\n\nWhat's new: the combination of graph-structured supervision with contrastive retrieval for exemplar selection. Random and semantic baselines don't cover it. The gains are consistent: up to 6.7 points average over BGE, much larger over random, with bigger gains on small models. The upper-bound analysis on Math23k is a good check. The distillation experiment in Section 4 is a nice practical extension and shows the method works without labeled graphs. They also honestly note that the distilled pairs may make the retriever rely on sentence-structure similarity rather than deep graph structure.\n\nWhere it's soft: evidence that the retriever aligns with graph structure outside Math23k is thin. Figure 4's scatter plot is on the training distribution, so it partly restates the training objective. The Calc-ape210k case study is qualitative and small. The alternative explanation — the retriever is a domain-adapted math-text retriever that finds semantically better examples rather than structurally analogous ones — stays plausible for the five non-Math23k datasets. The headline gain is driven by datasets with equation templates, where surface form likely correlates with structure; on gsm8k, math_qa, and aqua_rat the gains are small or absent. I don't think this sinks the paper. Even if the mechanism is partly domain adaptation, the method is still modular, generator-agnostic, and beats the semantic baseline on most datasets. But the paper overstates the case when it credits the gains specifically to structural analogy.\n\nSecond: all results are single runs, with no error bars, seeds, or significance tests. Some margins are under a point. Third: no code or data release is mentioned. Also, the retriever is trained on a translated subset of Math23k; translation details matter for the English datasets.\n\nAudience: people working on few-shot MWP, RAG for reasoning, or example selection. A solid contribution to that subfield, not a paradigm shift. Recommendation: send to peer review, with requests for error bars, an out-of-distribution graph-similarity check, and code release.","headline":"Computational-graph-based exemplar retrieval is a real, consistent win for few-shot math word problems, but the paper's mechanistic claim that graph analogy drives the gains is only directly validated on the training distribution.","tokens_in":13195,"tokens_out":3688,"would_cite":true,"duration_ms":31770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retriever that matches the arithmetic structure of math word problems improves few-shot exact-match accuracy by up to 6.7 points on average over semantic baselines, and up to 19.5 points over random selection, without retraining the…","keywords":["math word problems","few-shot prompting","computational graph","retrieval-augmented generation","contrastive learning","large language models","exact-match accuracy","analogy-based retrieval"],"falsifier":"A decisive check: on a graph-labeled test set, compare prompts built from top-k retrievals whose graphs match the query against prompts built from top-k retrievals whose graphs do not match but whose texts are equally similar; if exact-match accuracy is no higher in the graph-matched condition, the paper's central mechanism is not the cause of the gains.","tokens_in":12162,"feed_emoji":"🧮","tokens_out":5793,"duration_ms":48100,"temperature":0.7,"pith_summary":"This paper argues that the bottleneck in few-shot math word problem solving is not the generator but the choice of examples: randomly picked or semantically similar exemplars often share surface wording while differing in the arithmetic operations needed. The authors train a lightweight text retriever with contrastive learning, using questions that share the same computational graph as positive pairs, and then feed the top-k retrieved question-answer pairs into the prompt. Across six English and Chinese datasets and five generator sizes, this structural retrieval outperforms semantic retrieval by up to 6.7 points on average exact-match accuracy, and random selection by up to 19.5 points, without fine-tuning the generator. The result matters because it offers a modular, training-free way to strengthen LLM reasoning on math: give the model an analogous worked solution rather than a merely topical one.","feed_headline":"Retrieval by arithmetic structure lifts math few-shot scores by 6.7 points","feed_subtitle":"Choosing examples with matching reasoning paths beats semantic similarity and random picks on six datasets.","key_machinery":"The carrying object is the computational graph: a normalized arithmetic expression (e.g., $x=\\text{num}_a*\\text{num}_b/\\text{num}_c$) that abstracts away numbers, entities, and wording to capture the reasoning path of a math word problem. The mechanism around it is a contrastively trained retriever: an encoder with an added two-layer MLP pooler, trained with an infoNCE loss where positive pairs are questions with identical computational graphs and negatives are in-batch questions with different graphs. At inference the retriever computes top-$k$ similar questions from the corpus and their solutions are concatenated into the few-shot prompt. This machinery does the work of aligning example selection with the hidden structure of the target problem, without retraining or altering the generation model.","core_discovery":"The central claim is that analogy by arithmetic structure, not by topic, is what makes few-shot exemplars useful for math word problems. The paper represents each problem's reasoning path as a computational graph, a template such as $x=(\\text{num}_a - \\text{num}_b)/\\text{num}_c$, and trains an encoder-based retriever so that questions whose graphs are identical embed close together. At inference, the retriever selects the top-$k$ question-solution pairs from the corpus by embedding similarity and places them in the prompt; the generator is untouched. On Math23k, ape210k, gsm8k, math_qa, Calc-ape210k, and aqua_rat, this recipe improves exact-match accuracy over semantic retrieval across nearly all settings, with the largest gains on smaller generators, and approaches the upper bound obtained when the true graph itself is used for retrieval on Math23k.","pith_inferences":["Beyond the six datasets tested, the same graph-analogy retrieval could plausibly generalize to other structured reasoning tasks—such as symbolic logic, table-based arithmetic, or code—where a problem's 'reasoning path' can be normalized into an abstract template.","The paper's own observation that distilled data keeps sentence structure more similar than labeled data suggests an implicit risk: a text retriever trained on such pairs may latch onto syntactic templates rather than true graphs; one testable fix is to adversarially vary sentence structure more aggressively during distillation.","Because gains concentrate in smaller models, an implicit consequence is that structural retrieval is a cheap way to deploy capable math solving on edge devices, where large generators are impractical.","The upper-bound experiment implies the ceiling is not the generator but the retriever's ability to approximate graph similarity; further gains might come from combining graph-level supervision with more powerful encoders, not from larger prompts."],"forward_implications":["On five of the six datasets, choosing exemplars by computational-graph analogy improves exact-match accuracy over both semantic and random retrieval across generator sizes from 0.5B to 70B/72B.","The gain is largest for small generators, so structural retrieval can partially compensate for weaker reasoning in compact models.","Training on as little as 25% of Math23k (about 5,000 samples) is enough to reach stable retrieval quality, and 10% already yields a 4-point gain over semantic retrieval on aqua_rat.","The retriever still helps when the retrieval corpus is out-of-distribution: using ape210k as the corpus for SuperCLUE-Math6 test questions beats semantic and random selection.","LLM-distilled positive pairs generated by rewriting questions to keep the same graph can substitute for human-labeled graphs, outperforming semantic retrieval while slightly underperforming labeled training data."],"supporting_citations":[{"why":"supplies the Math23k training corpus whose computational graphs define the positive pairs for contrastive training","marker":"(Wang et al., 2017)"},{"why":"defines the infoNCE contrastive loss used to train the retriever","marker":"(Oord et al., 2018)"},{"why":"provides the base encoder that the retriever fine-tunes","marker":"(Xiao et al., 2023)"},{"why":"provides the LLaMA-3 generator models evaluated in the experiments","marker":"(Dubey et al., 2024)"},{"why":"supplies the gsm8k English evaluation dataset","marker":"(Cobbe et al., 2021)"},{"why":"supplies the ape210k Chinese evaluation dataset and retrieval corpus","marker":"(Zhao et al., 2020)"},{"why":"supplies the math_qa evaluation dataset","marker":"(Amini et al., 2019)"},{"why":"supplies the aqua_rat evaluation dataset","marker":"(Ling et al., 2017)"},{"why":"supplies the Calc-ape210k evaluation dataset","marker":"(Kadlčík et al., 2023)"},{"why":"provides the LLM used to distill training data for the retriever without human graph labels","marker":"(OpenAI et al., 2024)"}],"fun_headline_variants":["Retrieval by reasoning structure lifts math few-shot scores 6.7%","Math analogies: matching problem graphs beats semantic similarity","Few-shot math: retrieve by computation graph, not topic","Structural analogy boosts math word problem solving by 6.7%","Graph-based retrieval sharpens few-shot math reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a retriever trained on exact computational-graph equality within 25% of Math23k transfers structural similarity to five other datasets, so that top-$k$ text embeddings select exemplars whose hidden arithmetic structure matches the query rather than merely their wording.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval by reasoning structure lifts math few-shot scores 6.7%","Math analogies: matching problem graphs beats semantic similarity","Few-shot math: retrieve by computation graph, not topic","Structural analogy boosts math word problem solving by 6.7%","Graph-based retrieval sharpens few-shot math reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1161,"prompt_tokens":857,"completion_tokens":304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":473,"tokens_out":304,"duration_ms":3474,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:05:22.722769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: on a graph-labeled test set, compare prompts built from top-k retrievals whose graphs match the query against prompts built from top-k retrievals whose graphs do not match but whose texts are equally similar; if exact-match accuracy is no higher in the graph-matched condition, the paper's central mechanism is not the cause of the gains.","supporting_citations":[],"review_version":1}