{"id":"f3a49a04-2d05-4a05-a912-73357792afee","arxiv_id":"2505.13406","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"AutoMathKG constructs and auto-updates a natural-language mathematical knowledge graph using LLM augmentation, SBERT vector search, and a LoRA-tuned math reasoning model.","lead":"This paper builds an automatically updating map of mathematical knowledge, linking definitions, theorems, and problems in one searchable graph. It adds a vector search index and a math reasoning model to keep the map growing and complete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reachability claim is confounded: neighbor text is embedded into the query vectors and then used as the reachability ground truth; a no-refs rerun would settle it.","rationale":"The paper is a legitimate systems proposal: it constructs a 13k-entity graph, a vector store, and an LLM-based update pipeline, and the case studies in Section 8 show plausible behavior. The central quantitative evidence, however, is the reachability comparison, and that evidence is not trustworthy as a comparison against KG embeddings because the input representation contains the reference lists that define the ground-truth reachability labels. The reader's weakest_assumption identifies exactly this point, and I agree with it. The Math LLM evaluation is also thin (manual ratings, no baselines, no inter-annotator reliability), but I would not make it the primary attack because the abstract's first stated experimental claim concerns MathVD reachability, and that claim has a crisp computational test. If the no-refs rerun preserves Hits@q, the concern would be retired; if it collapses, the paper should either reframe MathVD as a text-similarity retriever with explicit reference-field features or evaluate against baselines that receive the same reference-text features. The verdict should remain CONDITIONAL: the system may work, but the empirical support for the headline comparison must be re-run and reported before acceptance.","tokens_in":24212,"tokens_out":4106,"duration_ms":42060,"concrete_test":"Re-run the Section 7.3.1 protocol with the same 100 sampled entities and the same k-hop ground truth, but embed each entity using only S1-S3 (title, field, contents) from Eq. 1, omitting in-refs and out-refs, under both the concatenation and weighted-sum variants. If Hits@10 and Hits@15 for these no-refs MathVD models fall to the TransE/BoxE range in Table 7, the reported superiority is an artifact of embedding the evaluation labels. As a sharper control, build a non-neural bag-of-words vector from neighbor titles alone and evaluate it under Eq. 4; if this trivial retriever matches or beats MathVD's Hits@q, the structural-search interpretation is fully explained by exact neighbor-text overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim is the reachability superiority of MathVD over the five KG embedding baselines (Section 7.3.1, Table 7), and this comparison is confounded by construction. In Section 5, MathVD1 concatenates S4(v) = 'in references: v[in refs]' and S5(v) = 'out references: v[out refs]' into the SBERT input (Eqs. 1-2), while MathVD2 weight-sums embeddings of the same five fields (Eq. 3). The evaluation metric, Hits@q (Eq. 4), is computed over k-hop reachability in the same directed reference graph that populates those in-refs/out-refs fields. For a two-hop query, the query vector and a candidate vector can both contain the title of the intermediate entity, so high cosine similarity can arise from literal text overlap in embedded adjacency lists rather than from any learned structural representation. The KG embedding baselines are trained on the graph but evaluated by cosine similarity of entity embeddings, a protocol that does not credit their trained scoring functions and cannot match this hard-coded neighbor-text advantage. Section 8.2's own ablation shows that removing reference fields degrades retrieval, and the examples in Table 13 show the top VD1-all hit is a direct textual consequence listed in the target's references. This does not make MathVD useless for fuzzy search or knowledge fusion, but it does mean the abstract's claim of 'superior reachability query results ... compared to five baselines' is not established as a test of structural search.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoMathKG, a natural-language mathematical knowledge graph built from ProofWiki, textbooks, arXiv papers, and TheoremQA, with vertices representing definitions, theorems, and problems and directed edges representing reference relationships. The construction pipeline uses rule-based extraction followed by Llama-2-7b in-context learning for augmentation, and the graph is stored in a JSON schema with three levels of information. For retrieval, MathVD constructs SBERT-based vector databases via two embedding strategies: MathVD1 concatenates title, field, contents, in-refs, and out-refs text, while MathVD2 weight-summarizes per-field embeddings. Automatic updates are handled by Math LLM for knowledge completion and by a MathVD-plus-LLM fusion mechanism for merging or adding entities. The main empirical claims are that MathVD achieves superior reachability-query performance over five knowledge-graph embedding baselines and that Math LLM demonstrates strong reasoning capability.","tokens_in":24587,"tokens_out":4183,"duration_ms":41311,"significance":"If the central claims held, the paper would provide a useful, practically deployable system: a broad-coverage, automatically updatable mathematical knowledge graph with an associated vector database and a reasoning assistant. The strengths are the explicit end-to-end pipeline, the detailed JSON entity schema, the inclusion of diverse sources, the documentation of ICL prompt templates, an ablation of the reference fields, external-query examples, and concrete case studies of knowledge completion and fusion. However, the headline quantitative claim about reachability queries is, by construction of the embeddings, confounded with direct textual overlap of neighbor titles, and the reasoning-capability evaluation lacks baseline comparisons and reliability metrics; both issues are load-bearing for the abstract's claims.","major_comments":[{"comment":"The reachability-query evaluation is circular with respect to the information embedded in the vectors. In both MathVD1 and MathVD2, the entity description includes S4(v) = 'in references: v[in refs]' and S5(v) = 'out references: v[out refs]', so the stored titles of neighboring entities are part of every vector. The Hits@q metric in Eq. (4) is then computed over k-hop reachability in the very same directed reference graph that fills these fields. For a query entity and a candidate entity that are k-hop reachable through an intermediate entity, the intermediate title can appear verbatim in both vectors, so high cosine similarity can result from literal name overlap in the embedded adjacency lists rather than from any learned structural representation. The ablation in Section 8.2 confirms that removing the reference fields degrades retrieval, and Table 13 shows that the top VD1-all hit for 'probability measure is subadditive' is 'Boole's inequality', an entity that is directly listed in the target's outgoing references. This means the claimed superiority over the five KG embedding baselines in Table 7 is not established as a test of structural search. I request a 'no-refs' evaluation of MathVD against the same baselines, or an evaluation on held-out edges, or an evaluation that uses the trained scoring functions of the baselines (rather than only cosine similarity of entity embeddings), to break the circularity.","section":"Section 5.1 (Eqs. 1-2), Section 5.2 (Eq. 3), Section 7.3.1 (Eq. 4), Section 8.2, Table 13"},{"comment":"The evaluation of Math LLM's reasoning capability is not sufficient to support the abstract's claim of 'robust mathematical reasoning capability'. The results consist of ratings assigned by the authors on a manually designed 1-5 scale, with no comparison baseline (e.g., the unmodified gemma-7b-it base model), no inter-annotator agreement measure, no significance test, and no calibration of the rating scale. The average ratings in Table 8 range from approximately 3.05 to 3.78, which the text interprets positively, but without a baseline or reliability evidence these numbers do not establish robustness. I ask for a direct comparison with the base model and with at least a standard instruction-tuned baseline, plus reporting of annotation protocol and agreement if the manual scoring is retained.","section":"Section 7.3.2, Table 8, Figure 10"}],"minor_comments":[{"comment":"There is a typo in the table caption: 'Courpus' should be 'Corpus'.","section":"Table 1"},{"comment":"The symbol q is used both for the number of retrieved neighbors and in the name 'Hits@q'; please define explicitly whether Hits@q is precision at q (r divided by the number of retrieved entities) and note that this differs from the 'hits at k' convention used in link prediction.","section":"Section 7.3.1, Eq. (4)"},{"comment":"The Kolmogorov-Smirnov test is mentioned but no test statistic or p-value is reported, so the claim that the two precision distributions show no significant difference cannot be verified.","section":"Section 7.3.3"},{"comment":"The attribute name is written as 'references tactics' in Fig. 4 and Table B3 but as 'references_tactics' in the JSON example in Fig. E1; please standardize the naming.","section":"Fig. 4 and Table B3"},{"comment":"No training details or hyperparameters for the PyKEEN baselines are given (number of epochs, negative sampling, embedding dimension is 384 but other settings are absent), which makes it difficult to assess whether the comparison is fair.","section":"Section 7.3.1, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The core construction pipeline and the KG itself appear to be real and potentially useful, but the reachability evaluation is the paper's headline contribution and it is confounded by the design of the embeddings. I would not be comfortable accepting until the authors rerun the evaluation without reference-text fields, on held-out edges, or with properly used baseline scoring functions. The reasoning evaluation is also weaker than the abstract implies. No concerns about authorship or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my take on AutoMathKG. The genuinely useful part is the integrated pipeline: they build a directed math KG of Definitions, Theorems, and Problems from ProofWiki, textbooks, arXiv, and TheoremQA; augment entities and references with Llama-2 via ICL; embed entities with SBERT into a vector database; and add two update mechanisms (a LoRA-finetuned Gemma math LLM for proof/solution completion, and vector-search plus LLM fusion for merging or adding entities). That integration is real work, and the graph statistics, JSON schema, and case studies give a concrete sense of what was built. The paper is worth reading for anyone constructing math KGs or retrieval-augmented reasoning for mathematics.\n\nThe soft spots are concentrated in the evaluation, and one is load-bearing. The reachability claim in Table 7 is confounded by construction. MathVD1 concatenates the in-references and out-references text into the SBERT input (Eqs. 1-2); MathVD2 does the same via weighted sums (Eq. 3). The evaluation metric is k-hop reachability over the same directed reference graph that populates those fields. So high Hits@q can be produced by literal title overlap in embedded adjacency lists rather than by any learned structural representation. The KG embedding baselines are evaluated by cosine similarity between entity embeddings, which does not even use their trained scoring functions. The paper's own ablation (Section 8.2) shows that removing reference fields degrades retrieval, and the Table 13 top hit is a direct consequence listed in the target's references. This does not make MathVD useless for fuzzy search, but the abstract's claim of superior reachability results compared to five baselines is not established as a fair test. Also, at Hits@1, TransE beats both MathVD variants, so the superiority is not even uniform on their own numbers.\n\nThe Math LLM evaluation is weaker: 234 manually rated questions, no baseline comparison, no inter-annotator reliability, and average ratings around 3 out of 5. That cannot support robust mathematical reasoning capability. The fusion case studies are illustrative only. No code or data is released; the KG is available on request, which limits reproducibility. The weight vector for MathVD2 and the number of fusion candidates are free parameters with no sensitivity analysis, but those are minor.\n\nOverall, the system proposal is coherent, the literature is cited honestly, and the limitations section is candid about scale choices. But the central empirical claims need rework. I would send it to review, because the construction is valuable and the confound is fixable: re-run reachability without embedding refs, compare against a proper link-prediction protocol, add a no-refs baseline for all methods, and include standard math LLM benchmarks or at least a base-model comparison. Would I cite it now? No. Bring it to the reading group? Maybe, because the circular evaluation is a useful teaching moment.","headline":"A useful system-building paper whose headline reachability claim is confounded because the query vectors contain the very reference lists used as ground truth; the rest is a reasonable proposal in need of reruns and baselines.","tokens_in":25037,"tokens_out":2418,"would_cite":false,"duration_ms":22880,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that mathematics can be represented as a directed graph of definitions, theorems, and problems that assembles and updates itself automatically from natural-language sources using large language models and vector-database…","keywords":["mathematical knowledge graph","vector database","LLM augmentation","in-context learning","fuzzy search","knowledge fusion","SBERT embeddings","automatic update"],"falsifier":"On a random sample of the test entities, recompute the 5-hop Hits@q for a MathVD variant whose vectors exclude the in-refs and out-refs fields and compare it against an oracle that simply matches the target's neighbour titles as text. If the title-matching oracle matches or beats MathVD while the no-refs variant collapses, the neighbour-embedding fields, not a learned structural representation, are producing the reachability result.","tokens_in":24013,"feed_emoji":"🧮","tokens_out":9151,"duration_ms":81789,"temperature":0.7,"pith_summary":"AutoMathKG proposes turning mathematics into a directed graph whose vertices are Definition, Theorem, and Problem entities and whose edges are reference relationships tagged with tactic roles such as premise, definition, and deduction. The authors argue that such a graph can be assembled automatically from ProofWiki, textbooks, arXiv preprints, and TheoremQA, with large language models augmenting entity content and extracting links through in-context learning. To make searching cheap, the graph is mirrored in a vector database built with two SBERT embedding strategies, and two update mechanisms keep the graph growing: an LLM-based 'Math LLM' completes missing proofs and solutions, while a vector search plus LLM judgement merges new entities into existing ones or adds them as fresh nodes. The value of the proposal, if it holds, is a wide-coverage mathematical resource that updates itself instead of being frozen at corpus-construction time.","feed_headline":"AutoMathKG builds a self-updating graph of mathematical knowledge","feed_subtitle":"Definitions, theorems, and problems become searchable entities that a math LLM keeps complete and merges automatically.","key_machinery":"The load-bearing object is the entity embedding that folds graph neighbourhoods into vector space. Each entity is turned into five descriptive sentences — title, field, contents, in-references, and out-references — and MathVD1 concatenates them before SBERT embedding, while MathVD2 embeds each field separately and takes a weighted sum with weights (0.5, 0.3, 0.1, 0.05, 0.05). The in-refs and out-refs sentences are the structural carriers: the paper's ablation shows that removing them degrades similar-entity retrieval, while keeping them lets the top retrieval for 'probability measure is subadditive' be Boole's inequality, a direct consequence recorded in the target's references. Around this embedding, the system layers a directed graph with nine tactic labels and two automatic-update mechanisms, one filling missing proofs via Math LLM and one merging or adding entities by LLM judgement over vector-search candidates.","core_discovery":"The central claim is that a mathematical knowledge graph can be built and sustained automatically when mathematics is represented as a directed graph of Definition, Theorem, and Problem entities connected by reference edges, with each entity stored in a JSON schema recording segmented content, tactic-labelled references, and incoming and outgoing neighbours. For similar-entity search, the paper embeds five descriptive fields — title, field, contents, in-refs, and out-refs — into SBERT vectors, either as one concatenated long text (MathVD1) or as a weighted sum of separately embedded fields (MathVD2). The neighbour fields, the paper argues, are what let cosine similarity retrieve entities that are structurally related rather than merely textually similar. The paper reports that both vector databases outperform five knowledge-graph embedding baselines on k-hop reachability queries, that human-rated precision for retrieving similar entities is around 95%, and that Math LLM, built from a 7-billion-parameter base with task adapters and self-calibration, produces mostly correct or correct-reasoning answers across six question categories.","pith_inferences":["I read the design as implying that any corpus with explicit reference environments can be absorbed without retraining the vector model, since the pipeline is rule-based extraction followed by LLM labelling; the update mechanism is source-agnostic in principle.","Because the reachability metric is computed over the same reference edges that are embedded in the neighbour fields, a sharper structural test would evaluate on held-out edges that were not present during vector construction; the paper does not report such a split.","The weighted-sum scheme suggests a tunable knob: shifting weight from contents toward neighbour lists should trade semantic relevance for structural proximity, and applications could calibrate the weights per task.","The stored tactic labels open an unexplored path to explanation generation, where a system walks a retrieved chain and tells the reader why each step follows — by premise, by lemma, or by deduction."],"forward_implications":["A mathematics knowledge base can be assembled from heterogeneous sources without manual curation; duplicates are detected by vector search and merged by LLM judgement.","Fuzzy search over mathematical statements becomes practical: a natural-language query like 'expectation in probability and statistics' returns relevant definitions, theorems, and problems in a single ranked list.","An LLM grounded in the graph can complete missing proof steps and problem solutions, so the resource grows while it is being used.","Because edges carry tactic labels, downstream systems can distinguish whether a reference functions as a premise, a definition, or a conclusion rather than treating all references as identical."],"supporting_citations":[{"why":"supplies the rule-based extraction method and the ProofWiki-derived base corpus for Definition and Theorem entities","marker":"[12]"},{"why":"provides the SBERT sentence embeddings used by both MathVD1 and MathVD2","marker":"[19]"},{"why":"is the LLM used for entity augmentation and all in-context-learning labelling","marker":"[18]"},{"why":"provides the LLM-annotated TheoremQA rationales that form Problem entities and the base fine-tuning data for Math LLM","marker":"[15]"},{"why":"supplies the original theorem-driven problems adopted as Problem entities","marker":"[27]"},{"why":"is the base instruction-tuned model from which Math LLM is fine-tuned","marker":"[49]"},{"why":"provides the low-rank adaptation method used to train the task adapters","marker":"[55]"},{"why":"is one of the five KG-embedding baselines compared for reachability and the one whose supervised one-hop optimization explains why MathVD trails only at q=1","marker":"[41]"}],"fun_headline_variants":["AutoMathKG: LLM-powered math graph that updates itself","Math graph auto-updates via LLM and vector similarity","AutoMathKG: definitions, theorems, problems auto-merged by LLM","Self-updating math KG: LLM fills proofs, merges duplicates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that embedding the list of an entity's reference neighbours measures structural reachability rather than simply letting a query match quoted neighbour titles verbatim; if exact title overlap is doing the work, the reported advantage over knowledge-graph embedding baselines would not demonstrate a general structural-search ability.","fun_headline_variants_meta":{"raw":{"variants":["AutoMathKG: LLM-powered math graph that updates itself","Math graph auto-updates via LLM and vector similarity","AutoMathKG: definitions, theorems, problems auto-merged by LLM","Self-updating math KG: LLM fills proofs, merges duplicates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1707,"prompt_tokens":1032,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":648,"tokens_out":675,"duration_ms":6300,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:13.496382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a random sample of the test entities, recompute the 5-hop Hits@q for a MathVD variant whose vectors exclude the in-refs and out-refs fields and compare it against an oracle that simply matches the target's neighbour titles as text. If the title-matching oracle matches or beats MathVD while the no-refs variant collapses, the neighbour-embedding fields, not a learned structural representation, are producing the reachability result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the original theorem-driven problems adopted as Problem entities"}],"review_version":1}