{"id":"374c4e1d-a708-43c8-999e-9bb12f1db12b","arxiv_id":"2507.10045","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"With explicit schema mappings and few-shot prompts, LLMs can translate SPARQL queries between DBpedia and Wikidata and between DBLP and OpenAlex with up to 86% accuracy.","lead":"The paper tests whether large language models can translate SPARQL queries between two knowledge graphs, DBpedia and Wikidata, and finds that with explicit schema mappings and example prompts, the best model translates 86% of queries correctly. A similar result is shown for the scholarly graphs DBLP and OpenAlex, suggesting LLM-based query translation could reduce manual work in knowledge graph integration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark curation filters out failing and non-comparable queries, so the reported 86% is an upper bound on operational translation accuracy; an uncurated test is needed.","rationale":"The central claim is that LLMs with structured prompting and schema-mapping tables offer a viable, scalable pathway to KG interoperability. For that claim to hold, the measured 86% must be indicative of performance on queries people actually need to translate. The benchmark construction undercuts this in a specific way: Section 3.1 filters the QALD-9-Plus train split to queries that execute on both KGs and return non-empty comparable results (371->240->100), and Section 3.2 keeps only DBLP templates the authors judged semantically translatable to OpenAlex, with gold queries manually written. These filters remove the hard tail of real-world translation demand—graph-specific idioms, failing queries, empty results, and cases requiring domain expertise—so the reported accuracies are conditional on a favorable selection. This is not an internal inconsistency in the experiments; the individual results are credible and the public code/data are a real asset. But the generalization from 'works on 100 curated pairs' to 'viable and scalable pathway' is exactly as strong as the representativeness of those 100 pairs. The proposed test directly estimates the selection effect: run the best configuration over the full QALD-9-Plus train set without filtering. If accuracy holds, the concern is resolved and the conditional acceptance can be upgraded. If it drops, the paper's headline must be reframed as an upper bound. Because the reader's conditional verdict already anticipates this risk, I recommend no change to the verdict, only that the revision address the uncurated benchmark or clearly scope the claim.","tokens_in":13438,"tokens_out":3828,"duration_ms":43299,"concrete_test":"Take all 371 English QALD-9-Plus train queries that have Wikidata SPARQL (including those that failed on DBpedia or returned empty), translate Wikidata→DBpedia with Mistral-Large-Instruct-2407 few-shot+ER mapping, execute on the same local snapshots, and score exact result match, counting non-executable/empty as incorrect. If accuracy falls more than 15 percentage points below 86%, the curated benchmark is not representative and the interoperability claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 keeps only queries that execute successfully on both DBpedia and Wikidata and return non-empty, comparable results, reducing 371 train queries to a curated 100; Section 3.2 similarly keeps only DBLP templates judged semantically translatable to OpenAlex, with gold OpenAlex queries manually authored. This selection removes exactly the cases where translation demand is real but hard: queries using graph-specific constructs, queries that fail on one graph, or queries that yield empty results. The abstract then generalizes the 86% Wikidata→DBpedia result into 'a viable and scalable pathway toward KG interoperability.' If the benchmark is not representative of real-world SPARQL translation difficulty, the headline accuracy is an upper bound, not an expected operational accuracy. A second, reinforcing issue is that every configuration was run once (Section 4.2), so the 86% has no variance estimate despite acknowledged LLM stochasticity (Section 7). The most load-bearing assumption is benchmark representativeness; the curated filter directly undermines the scalability claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether open-weight LLMs can translate SPARQL queries between heterogeneous knowledge graphs, focusing on DBpedia↔Wikidata and generalizing to DBLP→OpenAlex. The authors construct two 100-query benchmarks: one derived from QALD-9-Plus training queries that execute successfully on both DBpedia and Wikidata and return comparable non-empty results, and one derived from DBLP-QuAD templates deemed semantically translatable to OpenAlex, with manually authored gold OpenAlex queries. They evaluate Llama-3.1-8B, DeepSeek-R1-Distill-Llama-70B, and Mistral-Large-Instruct-2407 under zero-shot, few-shot, and chain-of-thought prompting, with and without explicit entity-relation mapping tables. The headline results are 86% accuracy for Mistral-Large-Instruct-2407 on Wikidata→DBpedia (few-shot) and 86% on DBLP→OpenAlex (few-shot), with markedly lower performance in the opposite direction and for smaller models. The paper also contributes a detailed error taxonomy and analyzes co-occurrence of error types. The central claim is that LLMs with structured prompting and explicit schema mappings offer a viable and scalable pathway toward KG interoperability.","tokens_in":13550,"tokens_out":3479,"duration_ms":41460,"significance":"If the reported results are representative of operational translation demand, this is a meaningful step toward automating cross-KG SPARQL translation, a relatively underexplored task. The paper is, to the best of my knowledge, the first systematic study of SPARQL-to-SPARQL translation with LLMs on multiple KG pairs. Strengths include the public release of code and data, the use of stable local KG snapshots for reproducibility, the inclusion of a generalization task beyond encyclopaedic KGs, and a structured error analysis that goes beyond simple accuracy reporting. The main caveat is that the benchmarks are curated in ways that may remove exactly the hard and realistic cases, so the headline accuracies are better interpreted as upper bounds on performance for uncurated real-world queries. This limits the strength of the scalability claim in the abstract, but the empirical methodology is otherwise careful and the findings are informative for future work.","major_comments":[{"comment":"The primary benchmark is constructed by keeping only QALD-9-Plus training queries that 'successfully executed on both Wikidata and DBpedia and returned non-empty, comparable results,' reducing 371 queries to a curated 100. This filter removes precisely the queries that would be hardest to translate in practice: those that fail on one graph, use graph-specific constructs, or return empty or non-comparable results. Consequently, the 86% accuracy reported in §5.1 is an upper bound on accuracy for an uncurated set of real-world translation requests, and the abstract's claim of a 'viable and scalable pathway toward KG interoperability' is stronger than the evidence supports. I request that the authors quantify how many of the 371 queries were excluded for each reason, present results on the full set if feasible, or explicitly reframe the claims as applying to the curated, executable subset.","section":"§3.1, Table 1"},{"comment":"The DBLP→OpenAlex benchmark was created by selecting 100 DBLP-QuAD query templates 'semantically translatable to OpenAlex' and manually creating the gold OpenAlex SPARQL queries. This introduces two sources of optimistic bias: queries for DBLP-specific constructs (e.g., BibTeX types) are excluded, and the gold standard is the authors' own manual translation, so the exact-match evaluation in §3.4 may penalize equally valid alternative translations that an independent expert would accept. The paper should report how many DBLP-QuAD templates were excluded and why, and ideally have the gold queries validated by a second annotator or compare a sample of LLM 'incorrect' outputs against an independent human judgement of equivalence.","section":"§3.2, §5.3"},{"comment":"All accuracy numbers are single-run point estimates. Section 7 notes the stochasticity of LLM outputs, but no variance is reported. For N=100, the standard error of an 86% accuracy is approximately 3.4 percentage points, and different sampling of the one run could produce materially different results. This is load-bearing because the paper's main contribution is the 86% figures, not a qualitative phenomenon. I recommend running each configuration at least three times with different random seeds (or temperature settings) and reporting mean and standard deviation, particularly for the Mistral-Large-Instruct-2407 few-shot results.","section":"§5.1, §5.3, §4.2"},{"comment":"The evaluation uses a strict exact-match criterion: an LLM-generated query is correct only if its executed result set precisely matches the gold answer set. This criterion can both under- and over-estimate translation quality. It underestimates when a translation is semantically correct but uses different variable names, different but equivalent triple patterns, or different FILTER formulations that yield the same results; it overestimates if the gold query itself is not the only reasonable interpretation of the natural language question. Given the DBLP gold queries are manually authored by the authors, I suggest supplementing exact-match with a secondary evaluation, such as manual inspection of a random sample of 'incorrect' translations to estimate the false-negative rate, or a relaxed equivalence criterion based on answer-set overlap.","section":"§3.4"}],"minor_comments":[{"comment":"The mapping process left 144 of 637 DBpedia terms unmapped; the paper says queries with unmapped terms were retained to test robustness, but it is not reported how many of the 100 benchmark queries contain unmapped terms and whether these queries correlate with lower accuracy. Adding such an analysis would strengthen the discussion of ER mapping coverage.","section":"§3.3"},{"comment":"The zero-shot baseline without ER mapping was applied only to Llama 3.1-8B and Mistral-Large-Instruct-2407, not to DeepSeek-R1-Distill-Llama-70B; clarify whether this was an intentional experimental design choice and whether it affects the comparability of model results.","section":"§4.2.2"},{"comment":"The figures are referenced but not shown in the text excerpt provided; the captions say 'Correctly Translated Queries' but do not indicate error bars or statistical significance. If error bars are not available, state this explicitly in the captions.","section":"§5.1, Figures 2–3"},{"comment":"The NLQ categorization is useful, but it is unclear whether the distribution of the 100 selected queries matches the original QALD-9-Plus train set. A brief comparison of category proportions would help assess representativeness.","section":"§3.5, Table 2"},{"comment":"The variable name 'er2' for the entity-relation mapping is not self-explanatory and appears without introduction; consider renaming it to 'schema_mapping' or a similar descriptive term throughout the paper and the provided JSON example.","section":"§4.2.1"},{"comment":"The error counts are aggregated over 12 configurations per direction, which conflates model and prompting effects. Reporting per-configuration error distributions in an appendix would make the error analysis more actionable.","section":"§5.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about benchmark representativeness is genuine and material: the curation in §3.1 and §3.2 removes exactly the queries that would be difficult in practice, so the reported 86% figures are not a reliable estimate of operational accuracy. The single-run evaluation compounds this by providing no sense of variance. These issues are fixable with additional experiments or by softening the claims, so I do not recommend rejection, but the revision must address them directly. The paper otherwise appears methodologically sound and the error analysis is a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first solid evaluation of LLMs translating SPARQL between KG pairs, and it ships code, data, and a genuinely useful error taxonomy. The 86% numbers are real for the curated benchmark, but the curation means they are upper bounds, not operational accuracy.\n\nWhat's new: nobody else, by the authors' literature review, has benchmarked SPARQL-to-SPARQL translation with LLMs directly. The paper's contribution is the benchmark assembly, the prompting comparison, and the error categorization. The reproducible findings that explicit entity-relation mappings and few-shot examples are worth tens of points, and that Wikidata-to-DBpedia is much easier than the reverse, are solid empirical results. Public data and code make verification possible.\n\nSoft spots, in rough order. First, benchmark representativeness. Section 3.1 keeps only queries that execute successfully on both graphs and return non-empty comparable results, cutting 371 training queries to 100. Section 3.2 keeps only DBLP templates judged semantically translatable and hand-authors the OpenAlex gold queries. That removes exactly the hard cases: graph-specific constructs, empty-result queries, and queries where translation demand is real. So the abstract's 'viable and scalable pathway' overstates what the evidence supports. The paper's own limitations section admits moderate size, English-only, and one generalization domain, but it doesn't connect the curation bias to the headline claim. Second, every configuration ran once, so 86% has no variance estimate; LLM stochasticity is acknowledged but not quantified. Third, exact result-set match is strict and can penalize semantically equivalent queries that return the same facts in different shapes. That is a known issue with execution-based evaluation, and it works against the LLMs here. There is also a direct internal conflict: the abstract says the 86% Wikidata-to-DBpedia result came from a Chain-of-Thought approach, while Section 5.1 says it was few-shot prompting. That must be fixed before publication.\n\nNone of this kills the paper. The findings are directional and plausible, and the assembled benchmark and error taxonomy are reusable. The citation pattern is honest—the self-cited datasets are earlier published resources, not outputs of this work.\n\nRecommendation: do engage, and send it to peer review. The right revision keeps the benchmark numbers as what they are—curated-case performance—and adds either an uncurated split or an explicit discussion of how selection changes expected accuracy, plus variance estimates from repeated runs. I would not cite the 86% as operational accuracy; I would cite it as a first controlled baseline.","headline":"A useful first benchmark for LLM-based SPARQL translation, with headline accuracy plausible as an upper bound; needs an uncurated test set and multi-run variance before the scalability claim is trusted.","tokens_in":14123,"tokens_out":1966,"would_cite":true,"duration_ms":21033,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper sets out to show that a large open language model, given an explicit schema-mapping table and a few worked examples, can translate SPARQL queries between different knowledge graphs, reporting 86% accuracy on two test pairs.","keywords":["SPARQL translation","knowledge graph interoperability","large language models","DBpedia","Wikidata","DBLP","OpenAlex","few-shot prompting"],"falsifier":"Take the full set of source queries before filtering—including those that fail on one of the two graphs or return empty results—translate each with the best configuration, and execute the outputs against stable snapshots. If accuracy on this unfiltered set falls well below 86%, or if the excluded queries concentrate the failures, then the reported accuracy is a property of the curated benchmark rather than of the general translation pathway.","tokens_in":13178,"feed_emoji":"🔀","tokens_out":9820,"duration_ms":107685,"temperature":0.7,"pith_summary":"SPARQL is written against one graph's schema, so a query for DBpedia rarely works on Wikidata or OpenAlex; this paper asks whether a large language model can do the translation automatically. It assembles two 100-query benchmarks—aligned DBpedia-Wikidata questions from QALD-9-Plus and DBLP-to-OpenAlex query pairs—and tests three open models with zero-shot, few-shot, and chain-of-thought prompts, with and without explicit JSON mapping tables. The strongest configuration, Mistral-Large-Instruct-2407 with few-shot examples and mapping tables, reaches 86% on both Wikidata-to-DBpedia and DBLP-to-OpenAlex. The paper's central claim is that this recipe—large model, structured prompt, explicit entity-relation mappings, in-context examples—is a practical route to knowledge-graph interoperability, while the direction asymmetry (Wikidata-to-DBpedia being much easier than the reverse) shows that mapping quality and identifier style matter.","feed_headline":"Large language models translate SPARQL queries with 86% accuracy","feed_subtitle":"Mapping tables and four examples lift Wikidata-to-DBpedia and DBLP-to-OpenAlex translation to 86%.","key_machinery":"The carrying mechanism is the prompt's 'er2' variable: a JSON mapping table pairing each source-graph IRI with its target-graph IRI, for instance mapping DBpedia's dbo:director to Wikidata's wdt:P57. For DBpedia and Wikidata the table is built automatically from owl:sameAs, owl:equivalentProperty, and owl:equivalentClass links; for DBLP and OpenAlex it is built from shared ORCID identifiers plus manually aligned relations. The prompt combines the natural-language question, the source SPARQL query, the two graph names, this mapping table, and an instruction to emit the final query between <sparql> tags. Four few-shot examples, chosen with sentence embeddings and clustering for diversity, are added in the strongest configuration. The mapping table is what does the work: removing it drops accuracy to near zero, while adding it lifts even the smallest model by tens of percentage points.","core_discovery":"On its own terms, the paper discovers that SPARQL-to-SPARQL translation is not a generic reasoning task the models solve from their pretrained knowledge; it is a lookup-plus-restructure task that succeeds when the prompt tells the model which source IRI corresponds to which target IRI. With that mapping supplied, the largest model translates 86 of 100 Wikidata queries into DBpedia queries whose executed answer sets exactly match the gold answers, and the same model translates 86 of 100 DBLP queries to OpenAlex when four examples are included. Without mappings or examples, accuracy collapses—one model scores 0% on DBpedia-to-Wikidata and 1% on DBLP-to-OpenAlex. The paper also finds that errors are dominated by structural mistakes that often cascade from a single wrong mapping, and that the simpler human-readable identifiers of DBpedia make it an easier target than Wikidata's numeric identifiers.","pith_inferences":["Because both benchmarks were pre-filtered to queries that execute successfully on the source and target graphs, the 86% figures describe an upper bound for curated, translatable queries; real workloads containing graph-specific constructs, failing queries, or deep domain expertise would likely score lower.","The method could be used to bootstrap mappings for a new graph pair: generate candidate mappings with an LLM, verify or correct a handful by hand, then use the corrected set as the mapping table for larger-scale translation.","The error taxonomy points to a self-correction loop: since a single wrong mapping often cascades into a structural error, running the generated query, comparing the error type, and feeding that diagnosis back into the prompt may recover many failures without retraining."],"forward_implications":["A practical recipe for cross-graph query translation is: use a large open LLM, supply a JSON entity-relation mapping table, and include four diverse examples in the prompt; no fine-tuning or per-graph training is required.","The same recipe transfers from encyclopaedic graphs to a specialised scholarly pair, so it is plausible for other graph pairs with comparable mappings, not just DBpedia and Wikidata.","Translation is not symmetric: Wikidata-to-DBpedia is consistently easier than DBpedia-to-Wikidata, so evaluation and tooling should treat direction as a first-class factor.","The strict exact-answer-set evaluation means the reported accuracy is a conservative measure: a translation that returns the same answers in different formatting or with extra variables would be marked incorrect even if semantically faithful.","Keeping the mapping table up to date becomes an ongoing requirement, since graph schemas evolve and stale mappings will produce the structural errors the paper catalogues."],"supporting_citations":[{"why":"Supplies the aligned DBpedia-Wikidata natural-language and SPARQL pairs from which the 100-query primary benchmark is filtered.","marker":"[15]"},{"why":"Supplies the 100 DBLP queries adapted into the DBLP-to-OpenAlex generalization benchmark.","marker":"[27]"},{"why":"Defines Wikidata's numeric item/property identifier scheme that successful translations must reproduce.","marker":"[1]"},{"why":"Defines DBpedia's human-readable IRI and ontology scheme that successful translations must reproduce.","marker":"[2]"},{"why":"The largest model tested; its few-shot and chain-of-thought configurations produce the reported 86% accuracies.","marker":"[31]"},{"why":"Provides the chain-of-thought prompting method used as one of the five prompting variants.","marker":"[34]"},{"why":"Selects the four diverse few-shot examples through sentence embeddings and K-Means clustering.","marker":"[33]"},{"why":"Supplies the 8B-parameter baseline used to show how much accuracy improves when mappings and examples are added.","marker":"[30]"},{"why":"Supplies the 70B reasoning-specialised baseline used to assess model scale and chain-of-thought effects.","marker":"[32]"},{"why":"Defines the OpenAlex target graph for the generalization task.","marker":"[17]"}],"fun_headline_variants":["SPARQL translation hits 86% only with explicit IRI mappings","Zero-shot SPARQL translation fails; few-shot mapping lifts it to 86%","SPARQL translation: 86% with explicit mappings, 0% without","SPARQL translation is mapping-driven, not a generic LLM skill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmarks contain only queries whose gold SPARQL already works on both graphs and returns non-empty comparable answers, so the reported accuracy is measured on a filtered, well-behaved slice of real translation demand rather than on all queries a user might want to move between these knowledge graphs.","fun_headline_variants_meta":{"raw":{"variants":["SPARQL translation hits 86% only with explicit IRI mappings","Zero-shot SPARQL translation fails; few-shot mapping lifts it to 86%","SPARQL translation: 86% with explicit mappings, 0% without","SPARQL translation is mapping-driven, not a generic LLM skill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001137,"raw_usage":{"total_tokens":4723,"prompt_tokens":948,"completion_tokens":3775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3691}},"tokens_in":564,"tokens_out":3775,"duration_ms":31010,"temperature":1.0,"reasoning_tokens":3691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:41:23.754660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the full set of source queries before filtering—including those that fail on one of the two graphs or return empty results—translate each with the best configuration, and execute the outputs against stable snapshots. If accuracy on this unfiltered set falls well below 86%, or if the excluded queries concentrate the failures, then the reported accuracy is a property of the curated benchmark rather than of the general translation pathway.","supporting_citations":[{"cited_title":"DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph","cited_arxiv_id":null,"evidence_quote":"Supplies the 100 DBLP queries adapted into the DBLP-to-OpenAlex generalization benchmark."},{"cited_title":"Mistral Large 2","cited_arxiv_id":null,"evidence_quote":"The largest model tested; its few-shot and chain-of-thought configurations produce the reported 86% accuracies."},{"cited_title":"Chain- of-Thought Prompting Elicits Reasoning in Large Language Models","cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting method used as one of the five prompting variants."},{"cited_title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks","cited_arxiv_id":null,"evidence_quote":"Selects the four diverse few-shot examples through sentence embeddings and K-Means clustering."}],"review_version":1}