{"id":"63832715-3d0d-4e09-868c-90591c625805","arxiv_id":"2501.08540","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage prompt-chaining framework with JSON serialization and graph pruning improves LLM-based semantic modeling of structured data over prior systems on three benchmarks.","lead":"This paper presents Knowledge Prompt Chaining, a framework that serializes ontologies and structured data into JSON and uses two chained LLM calls to generate semantic labels and semantic graphs for CSV, JSON, and XML sources. The authors report higher precision and recall than prior semantic modeling systems on three museum and football datasets, using only three rows per table.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Updated dscrm gold standard without rescored baselines makes claimed superiority unverifiable.","rationale":"The reader's weakest_assumption exactly identifies the load-bearing flaw: the dscrm gold standard was modified by the authors, and the baselines were not re-scored on the modified standard. The paper itself admits the modification (Section 5.1, Appendix A), so this is not an external suspicion but a self-reported limitation. The evaluation metrics are set-based, so any change to the target set directly influences scores. Without releasing the updated standard or rescoring baselines, the comparison cannot be reproduced or validated. This alone justifies rejection. No additional concern is needed; the core claim of outperforming prior methods is unsupported as presented. The reader's verdict of REJECT with moderate confidence is appropriate, and our stress-test does not change it.","tokens_in":113,"tokens_out":2826,"duration_ms":37934,"concrete_test":"Obtain the original dscrm gold standard from Taheriyan et al. [2] or Vu et al. [4] and compute Precision/Recall for all four methods (Taheriyan, Serene, PGM-SM, and the authors' outputs) against that original standard, using the same test split and preprocessed data. If the relative ranking or the margin of improvement changes by more than 2 percentage points, the claimed superiority is an artifact of the relabeling. Alternatively, re-run one baseline, e.g., PGM-SM with Serene's labeler, on the authors' preprocessed data and score it against both the original and updated standards to quantify the effect of the changed gold standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that Knowledge Prompt Chaining outperforms prior semantic modeling systems—rests entirely on comparing Precision/Recall scores against a common gold standard. Section 5.1 states: 'we have updated the semantic models in the ground-truth datasets to ensure that they are more reasonable and interpretable.' Appendix A details these updates for dscrm: removal of leaf nodes not tied to table headers, renumbering of multiple crm:E52_Time-Span nodes, context-dependent choice between crm:E21_Person and crm:E39_Actor, and split crm:E55_Type nodes. These changes alter the set of correct triples S in the Precision/Recall definitions of Section 5.4. Because both metrics are set-based—Precision = |S∩S*|/|S*|, Recall = |S∩S*|/|S|—changing S directly changes scores even for fixed predictions. The paper does not state that Taheriyan [2], Serene [3], or PGM-SM [4] were re-run against the updated standard; the reported baseline scores appear to be from the original publications. The updated standard is not released, and the original predictions of the baselines are not provided. Therefore, the reported gains (e.g., +5.9% precision on dscrm, Table 3) could reflect label edits rather than modeling ability. This is not an internal inconsistency in the method, but it is a fatal gap in the evaluation: the comparison is not apples-to-apples and cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Knowledge Prompt Chaining (KPC), a two-stage LLM prompt-chaining pipeline for semantic modeling of structured data. The method serializes the input data, the domain ontology, and a small set of known semantic models into JSON, incorporates these into system prompts, then chains a semantic-labeling step and a semantic-graph-building step, followed by a graph-pruning post-process. The evaluation uses three datasets (dscrm, dsedm, dsschema) and three LLMs (Claude 3.5 Sonnet, GPT-4 Turbo, DeepSeek-V2.5), comparing against Taheriyan, Serene, and PGM-SM. The paper claims semantic labeling precision of 96.9% or higher and semantic modeling gains of 5.9%, 4.6%, and 0.6% over prior methods, using only three rows of each table as input.","tokens_in":10087,"tokens_out":5010,"duration_ms":50900,"significance":"If the empirical comparison were valid, the framework would be interesting because it is training-free, few-shot, and uses reduced structured input while reporting high scores; the public code link is also a strength. However, the central comparison is compromised because the dscrm gold standard was modified by the authors (Section 5.1 and Appendix A) and the prior methods were apparently not re-evaluated on the updated standard. Since the Precision and Recall definitions in Section 5.4 are set-based comparisons against the gold standard, the reported improvements over baselines may reflect a relabeling of the target rather than modeling ability. The stress-test concern therefore lands: the headline claim is not verifiable from the submitted manuscript.","major_comments":[{"comment":"The authors state in Section 5.1 that 'we have updated the semantic models in the ground-truth datasets,' and Appendix A lists specific changes to dscrm: removal of leaf nodes not tied to table headers, renumbering of crm:E52_Time-Span nodes, context-dependent choice between crm:E21_Person and crm:E39_Actor, and split crm:E55_Type nodes. These changes alter the gold-standard set S in the Precision = |S∩S*|/|S*| and Recall = |S∩S*|/|S| definitions of Section 5.4. The paper does not state that Taheriyan, Serene, or PGM-SM were re-run against the updated standard; the Table 3 baseline numbers appear to be taken from the original publications. Consequently, the dscrm improvements in Table 3 (e.g., Precision 0.878 vs. 0.819 for PGM-SM) are computed against different gold standards for the two arms of the comparison, so the claimed superiority is not established. The updated standard is not released, and the original baseline predictions are not provided, so the comparison cannot be independently checked.","section":"Section 5.1, Appendix A, Table 3"},{"comment":"The experiments report point estimates only, with no standard deviations, confidence intervals, or significance tests, even though LLM outputs are stochastic and the experimental description mentions only two random seeds. The smallest claimed gain, 0.6 percentage points on dsschema in Table 3, is within the range one would expect from seed variation, so the robustness of the improvement over Serene (0.866 vs. 0.860 precision) is not demonstrated.","section":"Section 5.4, Tables 3-5"},{"comment":"The comparison with prior methods may also be mismatched in data preprocessing: the authors converted all table data to JSON, retained only three records per table, and replaced missing values with '<Empty>', but the paper does not report whether the published baseline scores were obtained on the same preprocessed inputs. Without this information, the 'reduced structured input' claim is not a controlled comparison, and even the absolute baseline scores cannot be taken as comparable.","section":"Section 5.3 and Section 5.1"}],"minor_comments":[{"comment":"There are several typos and wording errors: 'insturction' in the abstract, 'sterilizes' in Section 1 (should be 'serializes'), 'Chian2' in Section 1, and 'prat' in Section 6.3 (should be 'part').","section":"Abstract and Section 1"},{"comment":"The tables lack descriptive captions; for example, Table 2 is introduced only as a comparison of semantic labeling methods but its caption does not state the metric and dataset details. Adding captions that specify the metric, dataset, and model for each table would improve readability.","section":"Tables 2-6"},{"comment":"Figure 2 gives a schematic of the prompt templates, but the exact system prompts and the precise JSON serialization format are not included. Since the method is prompt-based, releasing the full prompts or pointing to a versioned artifact would be important for reproducibility.","section":"Figure 2 and Section 4.2"},{"comment":"The numbering rules for crm:E52_Time-Span nodes are not fully machine-checkable: when multiple such nodes remain after the birth/death pairs are numbered, the paper does not specify the order in which the remaining nodes are numbered.","section":"Appendix A"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the modified dscrm gold standard combined with un-re-run baselines; this prevents the central claim from being assessed. I would not rule out a future submission if the authors release the updated standard, re-evaluate all baselines on it, and report variance over repeated runs, but the current manuscript is not publishable as a comparison study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before deciding whether to spend time on it. First, the method itself is a sensible, no-training combination of JSON serialization, knowledge prompts, chain-of-thought, and a two-step chain with graph pruning. Second, the central empirical claim—that it beats the prior best systems—is undercut by the authors' own disclosure that they revised the dscrm ground truth and did not re-run the baselines on the revised standard.\n\nWhat is new: the paper applies LLM prompt chaining to semantic modeling of structured data, a task previously dominated by Steiner-tree, constraint, and probabilistic-graph approaches. The two-stage chain (labeling, then graph building) plus connectivity-based pruning is a simple but reasonable extension of known ingredients, and the reported results on dsedm and dsschema are not obviously compromised. The ablation study (Table 7) shows that prompting chaining and pruning each contribute, which is real evidence that the design matters. The idea that three rows per table can suffice for LLM-based modeling is worth testing in other settings.\n\nWhere the paper is soft: the dscrm comparison is not apples-to-apples. Section 5.1 and Appendix A say the ground-truth semantic models were updated by the authors, with specific changes like removing leaf nodes, renumbering E52_Time-Span nodes, and splitting E55_Type nodes. The Precision/Recall formulas treat the gold standard as the set of correct triples. If the baselines were scored on the original standard and the authors' method on the updated one, the +5.9% precision gain on dscrm could partly reflect relabeling rather than modeling ability. The paper does not state that Taheriyan, Serene, or PGM-SM were re-run on the updated standard, and the updated standard is not released. This is a load-bearing flaw for the headline claim. Also, the paper lacks error bars and significance tests across the random seeds, and the prompt templates are only summarized in a figure, not fully legible. The baselines' original scores are adopted without re-evaluation, which compounds the comparison issue.\n\nOn the other hand, the stress-test note is fair but not fatal to the whole paper. The dsedm and dsschema results, the ablation, and the token-efficiency point stand on their own. The method is not presented as a formal derivation; it is an empirical pipeline, and the evaluation is what fails to convince.\n\nThe paper deserves a serious referee despite the evaluation gap. A referee could demand a re-run of baselines on the updated dscrm standard, release of the updated gold standard, and error bars. The core prompting idea is plausible, and the authors are transparent about the update, which suggests honest engagement with the data. It is not a desk reject; it is a revise-and-resubmit candidate after the comparison is fixed.","headline":"A credible prompting pipeline for semantic modeling, but the dscrm gold-standard edit without rescored baselines makes the headline gains unverifiable.","tokens_in":10575,"tokens_out":722,"would_cite":false,"duration_ms":9106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge Prompt Chaining claims to beat established semantic-modeling systems using only serialized ontologies, a few rows, and a two-step LLM prompt chain.","keywords":["semantic modeling","semantic labeling","prompt chaining","large language models","in-context learning","ontology alignment","structured data","knowledge graphs"],"falsifier":"Re-score the three baseline systems on the updated dscrm gold standard, applying the numbering and node-replacement rules in Appendix A with the same half-shot data splits, and check whether the reported 5.9% precision improvement over the prior best on dscrm survives.","tokens_in":9582,"feed_emoji":"🧩","tokens_out":5881,"duration_ms":53382,"temperature":0.7,"pith_summary":"The paper attempts to show that semantic modeling—mapping the columns of CSV, JSON, or XML files to classes and properties of a domain ontology, then linking them into a semantic graph—can be done by a no-fine-tuning chain of LLM prompts. Rather than building an integrated graph or training a dedicated model, the framework serializes the ontology and a few sample rows into a system prompt, first asks the LLM to label each attribute, then asks it to build the graph from those labels. On three real-world datasets it reports semantic-labeling precision of 96.9% or higher and semantic-modeling gains of 5.9%, 4.6%, and 0.6% over the best prior methods, using only three rows per table as input. The reader would care because it suggests that expensive, task-specific machinery for schema integration can be replaced by prompt engineering plus a graph-pruning step.","feed_headline":"No-training prompt chain beats prior semantic modeling systems","feed_subtitle":"Three rows of data plus a serialized ontology let LLMs label columns and build semantic graphs with 92.9% recall.","key_machinery":"The central object is the Knowledge Prompt Chaining pipeline: a system prompt that serializes the domain ontology and known semantic models into JSON, a two-step chain of prompts (semantic labeling, then graph building) that requires the LLM to give reasoning before each answer, and a graph-pruning step that deletes predicted nodes not linked to any attribute in the input. The ontology serialization supplies the structure and latent relationships of the graph; the chaining decomposes the task into the same two steps the gold-standard construction uses; the reasoning requirement works as a chain-of-thought; the pruning catches hallucinations by constraining outputs to the ontology-data connection.","core_discovery":"The central claim is that a two-step prompt chain, with the ontology and example semantic models injected as serialized JSON in the system prompt, outperforms established automatic semantic-modeling methods even when given a fraction of the data those methods need. The first step generates semantic labels for each attribute; the second step feeds those labels back to the same model to construct a semantic graph, with a final pruning pass that removes hallucinated nodes that cannot connect to any attribute. The paper reports that this reaches 96.9% or higher semantic-labeling precision and 86.6% to 92.9% recall on the resulting semantic graphs, exceeding the reported scores of prior Steiner-tree, constraint-programming, and probabilistic-graph approaches on all three datasets.","pith_inferences":["Editorial inference: if the framework's gains generalize, the same two-step serialization-plus-chain recipe could be applied to related integration tasks such as schema matching and knowledge-graph population, where a prompt chain could replace trained matchers.","Editorial inference: the reported dependence on the updated dscrm gold standard means the headline comparison may partly reflect the relabeling; a direct re-run of prior baselines on the Appendix A rules would settle how much of the gain is architectural rather than due to the changed target.","Editorial inference: the framework's performance likely rests on the LLM's prior exposure to the domains and ontologies involved; on obscure or private ontologies, the three-row sample and expert rules may not be sufficient, so a stress test in a novel domain would be informative."],"forward_implications":["If the reported results hold, automatic semantic modeling can be done with no task-specific model or graph construction, relying instead on a general-purpose LLM and prompt design.","The two-step chain plus pruning yields a per-source runtime under 30 seconds in the reported settings, making the approach practical for interactive use.","Because only three rows are used per source, the approach cuts token usage relative to methods that consume entire tables, which could lower deployment cost.","The method works across three different ontologies and data formats, suggesting the pipeline transfers across domains."],"supporting_citations":[{"why":"Supplies the first semantic-modeling baseline (Steiner-tree search) that the framework is compared against.","marker":"[2]"},{"why":"Supplies the constraint-solving baseline (Serene) used in both semantic-labeling and semantic-modeling comparisons.","marker":"[3]"},{"why":"Supplies the probabilistic-graph baseline (PGM-SM) used in the semantic-modeling comparison.","marker":"[4]"},{"why":"Motivates the choice of JSON as the text serialization for tabular data, which the framework uses for tables and ontology.","marker":"[10]"},{"why":"Provides the chain-of-thought technique that the framework adapts by asking for reasoning before each output.","marker":"[14]"},{"why":"Supplies the survey basis for text-based serialization of structured data for LLMs.","marker":"[20]"}],"fun_headline_variants":["No-training prompt chain beats semantic modeling baselines","Prompt chain learns semantics from just 3 data rows","LLM prompt chain auto-labels data and maps ontologies","Knowledge prompt chaining: semantic models from sparse input"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains depend on the assumption that the updated gold-standard used to score the new method is the same target the baselines were scored against; the paper updates the dscrm ground truth without releasing it or re-running the baselines on it.","fun_headline_variants_meta":{"raw":{"variants":["No-training prompt chain beats semantic modeling baselines","Prompt chain learns semantics from just 3 data rows","LLM prompt chain auto-labels data and maps ontologies","Knowledge prompt chaining: semantic models from sparse input"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2575,"prompt_tokens":842,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":1669}},"tokens_in":458,"tokens_out":1733,"duration_ms":14259,"temperature":1.0,"reasoning_tokens":1669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:23:40.911855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the three baseline systems on the updated dscrm gold standard, applying the numbering and node-replacement rules in Appendix A with the same half-shot data splits, and check whether the reported 5.9% precision improvement over the prior best on dscrm survives.","supporting_citations":[{"cited_title":"Knoblock, Pedro A","cited_arxiv_id":null,"evidence_quote":"Supplies the first semantic-modeling baseline (Steiner-tree search) that the framework is compared against."},{"cited_title":"Machine learning and constraint programming for relational-to-ontology schema mapping","cited_arxiv_id":null,"evidence_quote":"Supplies the constraint-solving baseline (Serene) used in both semantic-labeling and semantic-modeling comparisons."},{"cited_title":"Learning semantic models of data sources using probabilistic graphical models","cited_arxiv_id":null,"evidence_quote":"Supplies the probabilistic-graph baseline (PGM-SM) used in the semantic-modeling comparison."},{"cited_title":"Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of JSON as the text serialization for tabular data, which the framework uses for tables and ontology."}],"review_version":1}