Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Structural Code Search using Natural Language Queries

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims natural-language queries can be translated into structural code search DSLs by an LLM with retrieval-augmented examples, reaching 55–70% precision and recall on a new 400-query Java benchmark.

desk verdict Useful benchmark and a clear LLM+RAG pipeline, but the shared synthetic generation of training and test queries makes the headline numbers optimistic. read the letter →

arxiv 2507.02107 v1 pith:T62EIMP2 submitted 2025-07-02 cs.SE cs.PL

Ben Limpanukorn , Yanjun Wang , Zach Patterson , Pranav Garg , Murali Krishna Ramanathan , Xiaofei Ma , Anoop Deoras , Miryung Kim This is my paper · ORCID
classification cs.SEcs.PL
keywords structuralcodesearchnaturallanguagequeryLLMtranslationretrieval-augmentedgenerationSemgrepGQLbenchmarkJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that developers can search code structurally without learning a domain-specific language, by having a large language model translate ordinary English queries into the DSL of a structural search engine. The proposed pipeline combines retrieval-augmented generation with automatically generated pairs of natural-language and DSL queries, then refines malformed DSL output through error feedback. On a new benchmark of 400 queries over 10 Java projects, the approach achieves precision and recall of 55–70%, and it outperforms semantic embedding search by up to 57% and direct LLM retrieval by up to 14% in F1. If this result holds, structural code search becomes accessible to developers who do not know Semgrep, GQL, or similar query languages.

What carries the argument

The load-bearing mechanism is an algorithm that enumerates structural search queries by starting from real code constructs and iteratively specializing or generalizing DSL queries until a target complexity is reached, then uses an LLM to convert each DSL query into an equivalent natural-language query. This produces the paired (NL, DSL) examples that populate the retrieval-augmented generation index and form the evaluation benchmark. The second key component is query refinement: a static analysis of the generated DSL query determines its target code-construct type, and if that type differs from the construct type identified in the natural-language query, the LLM is re-prompted with that feedback to regenerate the DSL query.

What would settle it

Collect a set of structural search requests written by working developers and run the NL-to-DSL pipeline on them; if precision and recall fall well below the 55–70% range, the synthetic benchmark queries do not represent real user phrasing.

Watch

Extended reading notes

Core claim

The central claim is that an LLM can reliably translate natural-language structural code search queries into executable DSL queries, provided it is given a few retrieved examples of paired (NL, DSL) queries and a feedback loop that compares the target code-construct type of the NL query with the construct type of the generated DSL query. The authors instantiate this approach for two DSLs: Semgrep, which matches code patterns over abstract syntax trees, and GQL, a graph-based DSL operating on a program dependence representation. Their evaluation reports 58.5% F1 on the GQL-derived benchmark and 70.0% F1 on the Semgrep-derived benchmark, with the approach beating a pure-LLM baseline by 6–14% F1 and a vector-search baseline by 55–57% F1, while using roughly 22 times fewer tokens than the LLM baseline.

Load-bearing premise

The benchmark's natural-language queries were generated by an LLM from structured templates rather than collected from real developers, so the measured translation accuracy may not match how developers actually phrase structural search requests.

Editorial extensions

If this is right

  • Developers could issue structural search queries in plain English and get precise, executable results for bug finding, refactoring, and code navigation without learning a DSL.
  • The translation approach is DSL-agnostic in principle, so it could be instantiated for other structural search languages such as CodeQL or Comby if paired (NL, DSL) training data is generated the same way.
  • The pipeline is far more token-efficient than asking an LLM to answer a search query directly, which makes it more feasible for large code corpora.
  • The new 400-query benchmark with natural-language and DSL query pairs can serve as a shared evaluation set for future structural code search systems.
  • Combining multiple NL-to-DSL engines with a router is suggested by the cross-benchmark results, since GQL and Semgrep support different code constructs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the synthetic NL queries, generated from templates and DSL structure, may be cleaner and more explicit than real developer phrasing; a router or query-normalization step could be tested against real usage logs.
  • The query-refinement loop only checks the top-level target construct type, so errors inside nested predicates could slip through; extending feedback to verify full predicate structure is a natural next test.
  • Because the benchmark excludes disjunctions and negations of code constructs, the reported 55–70% range likely bounds conjunctive queries only; extending the generator to those operators would show whether the approach's robustness carries over.
  • The large gap between vector search and NL-to-DSL suggests that hybrid systems, which use embedding retrieval to narrow candidate files before structural matching, could be a practical deployment pattern.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an approach for structural code search from natural language queries: an LLM with retrieval-augmented generation (RAG) translates an NL query into a DSL query (Semgrep or GQL), executes it with the DSL's static analysis engine, and optionally refines malformed translations via static-analysis-based error feedback. To support training and evaluation, the authors design an algorithm that enumerates DSL queries from a Java corpus and pairs each with an LLM-generated NL paraphrase, producing 200-query GQL and Semgrep benchmarks over 10 Java projects. Experiments report line-level F1 of 58.5% for GQL and 70.0% for Semgrep, with reported improvements over an NV-Embed vector search baseline (up to 57 points) and over a pure-LLM in-context retrieval baseline (up to 14 points), along with ablations showing the importance of paired (NL, DSL) RAG examples and query refinement.

Significance. If the results transfer to real usage, the work would be a useful step toward lowering the barrier to structural code search and would provide a first benchmark for NL-based structural search. The paper's strengths include a general, DSL-agnostic generation algorithm; detailed instantiation for two structurally different DSLs; a cross-benchmark analysis (Table 5) that exposes each DSL's limitations; a token-efficiency comparison that quantifies a practical advantage over pure-LLM retrieval; and an ablation table (Table 4) that clearly shows the decisive role of paired examples and inline comments. The central weakness is that the evaluation queries are synthetic and are produced by the same generation procedure as the RAG examples; without evidence on human-written queries, the external validity of the claimed 55-70% precision/recall is uncertain.

major comments (3)
  1. [§3.2, §5.1, §6.5] The main evaluation is subject to a distributional coupling between training and test. The benchmark's NL side is generated by prompting an LLM to translate DSL queries into natural language using structured descriptions of DSL predicates and a small set of corrected examples (§3.2, Figure 2); the RAG index used for translation is populated by the same generation algorithm (§3.1-§3.2, Table 4). The model is therefore evaluated on paraphrases produced by the same pipeline it was trained on, and the ablation in Table 4 shows that the paired examples are the decisive ingredient (F1 jumps from 2.4% to 33.5% when they are added). Section 6.5 addresses corpus disjointness and exact-duplicate filtering, but it does not address the more serious overlap in query phrasings, complexity distribution, and vocabulary between the generated benchmark and the generated RAG examples. This is load-bearing for the abstract's claim that developers can search structurally without learning a DSL. I would like to see either (a) an evaluation on queries written by developers, e.g., from a small user study or from developer search logs, or (b) a clearly framed claim that the benchmark measures round-trip consistency on a synthetic distribution, with a human judgment sample to estimate the gap.
  2. [§5.3] The match criterion is start-line equality only. Two code spans that begin on the same line but have different end lines are counted as equivalent, and a correct match whose start line differs, for instance because the query matches a multi-line construct and the ground truth records the first line of the enclosing statement, is counted as incorrect. Since many Semgrep patterns in the benchmark use pattern-inside clauses and multi-line patterns (§4.2, Listing 4), this metric can materially distort both precision and recall. The paper should report the sensitivity of Table 2 and Table 3 to an end-line-aware or full-span match criterion, and should state how ground-truth spans are normalized.
  3. [§6 and Table 3] The quantitative comparisons are reported as point estimates from what appears to be a single run, with no confidence intervals or significance tests. Given the stochasticity of LLM translation and the small number of queries (200 per benchmark, 10 in the Lite subsets), the claim that the proposed approach 'significantly outperforms' baselines is not statistically supported. At minimum, repeated runs with different seeds or temperature settings are needed for the main comparisons, and the Lite-benchmark comparisons should be interpreted with this limitation in mind.
minor comments (5)
  1. [§3.2] The number of human-verified few-shot examples and the verification process are not specified; please provide these details for reproducibility.
  2. [§5.1] The 10 Java projects are not identified; for a benchmark that is intended to be open-sourced, list project names or unique identifiers so others can reproduce the corpus.
  3. [§5.3] The metric definition should clarify whether precision and recall are macro-averaged per query or micro-averaged over all matches, and how queries with zero predicted or zero ground-truth matches are handled in the average.
  4. [§6.2] The vector search comparison is reported only at method granularity; since vector search cannot return line-level spans, an additional line-level upper bound derived from method-level retrieval would make the comparison more informative.
  5. [Algorithm 1] The notation `continue←↪` in lines 6, 15, and 16 is unconventional and should be replaced with a standard `continue` statement or a clear equivalent.

Circularity Check

1 steps flagged · score 6.0 of 10

The benchmark's NL queries are synthesized from the gold DSL queries using the same generation pipeline that builds the RAG few-shot bank, so the headline F1 is partly a round-trip consistency measure on a synthetic distribution.

  1. self definitional [Section 3.2, paragraph beginning 'Note that by generating DSL queries...'; instantiated in Sections 4.1-4.2 and acknowledged in Section 6.5.]
    "Note that by generating DSL queries (as described in Section 3.1) and then pairing these queries with equivalent NL queries, we obtain paired (NL, DSL) queries that are indexed in the RAG used by the LLM to translate NL query to DSL. Further, these (NL, DSL) pairs also serve as a benchmark dataset that we use for evaluation."

    The benchmark's natural-language side is produced by prompting an LLM to verbalize the very DSL query that is later used as ground truth, and the RAG few-shot examples are generated by the same DSL-query enumeration and NL-pairing procedure. Each test input is therefore a paraphrase of its own gold output, drawn from the same template/description distribution as the retrieval-support examples. The reported 55-70% precision/recall is an empirical score, not a forced value, but it is not evidence about arbitrary developer phrasing: it measures how well the model inverts the paper's own synthetic NL-generation pipeline.

full rationale

I found no load-bearing self-citation chain: GQL, Semgrep, and the baseline models are external, and no uniqueness theorem is imported from the authors' prior work. The central circularity is in benchmark construction. Sections 3.1-3.2 build both the RAG index and the test set from the same algorithm; Section 5.1 states that the code corpora and queries are disjoint, but that does not make the NL-query distribution independent of the generation pipeline; Section 6.5 concedes the shared generator and filters duplicates only at the DSL-query level. The translation task is therefore an inversion of the benchmark's own generation process: the NL input is constructed from the DSL output. This inflates the measured F1 relative to a benchmark collected from real developers, though it does not make the result tautological, because the LLM still must produce executable DSL and retrieval is measured against actual ground-truth code matches, and the comparisons against embedding and LLM baselines carry independent information. Score 6 reflects partial, not total, circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about the correctness of DSL engines, the faithfulness of LLM-generated NL queries, and the representativeness of the synthetic benchmark. No new physical or conceptual entities are invented. The main hand-chosen design choices are the complexity bounds and sampling weight; exact values are not reported.

free parameters (3)
  • Complexity bounds c_min, c_max
    Algorithm 1 takes minimum and maximum query complexity as inputs; the paper does not state the values used to generate the 400-query benchmarks, so the diversity and difficulty of the dataset depend on hand-chosen thresholds.
  • Vector search similarity threshold T = 0.25, 0.5, 0.75
    Used to evaluate the semantic search baseline; the choice of threshold strongly affects baseline recall and precision and is not part of the proposed method, but it shapes the reported comparison.
  • Construct type sampling weight 1/(1+c_e) = 1/(1+c_e)
    The near-uniform bias is a hand-chosen function; the paper does not evaluate sensitivity to this weighting.
assumptions (4)
  • domain assumption Semgrep and GQL execution engines correctly implement the generated DSL queries and return all syntactic matches.
    The ground truth for every benchmark query is the output of the DSL engine. If an engine misses matches or has bugs, the precision and recall numbers are affected. The paper does not validate engine outputs against a second oracle.
  • domain assumption The LLM-generated natural language queries are faithful, unambiguous restatements of the DSL queries.
    NL queries in the benchmark are produced by prompting an LLM with DSL-structured descriptions; no human verification beyond a few prompt examples is reported, so the benchmark's NL side may not represent real developer phrasing.
  • domain assumption Matching a code span by its start line is a valid equivalence criterion for code matches.
    The metrics count a predicted match as correct if its start line equals the ground-truth start line; this ignores multi-line spans and could both over- and under-count matches. Section 5.3.
  • ad hoc to paper The 10 selected Java projects and 400 generated queries are representative of real structural code search workloads.
    Projects are selected by license and size only; queries are synthetic, so external validity is assumed. Section 5.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structural Code Search using Natural Language Queries." pith.science (2026). https://pith.science/paper/T62EIMP2

@misc{pith2026250702107,
  author       = {Pith},
  title        = {Pith review of: Structural Code Search using Natural Language Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T62EIMP2}},
  note         = {Machine review of arXiv:2507.02107}
}
read the original abstract

Searching code is a common task that developers perform to understand APIs, learn common code patterns, and navigate code. Currently, developers most commonly search using keywords and regular expressions that are easy to use and widely available. Beyond keywords and regular expressions, structural code search tools allow developers to search for code based on its syntactic structure. This has numerous applications ranging from bug finding to systematically refactoring code. However, these structural code search tools operate on queries expressed in domain-specific languages (DSL) that can be difficult to learn and write. We propose to allow developers to use natural language to search for code structurally. Expressing queries in natural language provides an intuitive way to search for code and lowers the barrier to entry. In this work, we develop a novel general approach that combines the reasoning capabilities of an LLM to interpret natural language search queries with the power of structural search tools to efficiently and accurately retrieve relevant code. We then instantiate this approach for two structural code search DSLs: Semgrep and GQL. In our evaluation, we construct a new benchmark for structural code search consisting of 400 queries over 10 Java projects. We show that our approach for structural code search based on translating NL queries to DSL queries using an LLM is effective and robust, achieving a high precision and recall ranging from 55% - 70%. Further, our approach significantly outperforms baselines based on semantic code search and LLM retrievals by up to 57% and 14% on F1 scores.

Figures

Figures reproduced from arXiv: 2507.02107 by the authors.

Figure 1
Figure 1. Proposed approach for structural code search for queries posed in natural language. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Prompt used to translate queries expressed in a DSL to natural language. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Our approach provides the LLM with feedback to refine the query using static analysis of the generated [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Distributions of query complexity and the occurrence of code constructs in the GQL-Full and Semgrep [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The LLM prompt used to extract properties of the target code construct to instantiate in a GQL [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Example prompt used for In-Context Retrieval baselines. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Example of a complex and a simple NL query that is accurately matched to code by our structural [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Example of a complex and a simple NL query that is accurately matched to code by the NL-to-Semgrep [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Performance on search queries comprising different code entities (NL-to-Semgrep on the left and [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Performance on queries comprising up to 5 DSL constructs. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Code search query that is accurately answered by NL-to-GQL, but for which LLM baseline fails to [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

  1. [1]

    Amazon Web Services

    Inc. Amazon Web Services. 2024. AI For Software Developement - Amazon Q Developer - AWS. https://aws.amazon. com/q/developer/

    work page 2024

  2. [2]

    Sushil Krishna Bajracharya and Cristina Videira Lopes. 2010. Analyzing and mining a code search engine usage log. Empirical Software Engineering 17, 4–5 (Sept. 2010), 424–466. https://doi.org/10.1007/s10664-010-9144-6 Structural Code Search using Natural Language Queries 21

    work page doi:10.1007/s10664-010-9144-6 2010

  3. [3]

    Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. 2019. When deep learning met code search. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA...

    arXiv 2019

  4. [4]

    Inc. Codeium. 2024. Codeium - AI Code Completion and Chat. https://codeium.com/

    work page 2024

  5. [5]

    CodeQue.co. 2024. Multiline & Structural Code Search. https://marketplace.visualstudio.com/items?itemName= CodeQue.codeque

    work page 2024

  6. [6]

    Luca Di Grazia and Michael Pradel. 2023. Code Search: A Survey of Techniques for Finding Code. ACM Comput. Surv. 55, 11, Article 220 (Feb. 2023), 31 pages. https://doi.org/10.1145/3565971

    doi:10.1145/3565971 2023

  7. [7]

    Inc. Github. 2024. CodeQL overview - CodeQL. https://codeql.github.com/docs/codeql-overview/

    work page 2024

  8. [8]

    Inc. Github. 2024. Github Copilot. https://github.com/features/copilot

    work page 2024

Show all 25 references
  1. [9]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv:1909.09436 [cs.LG] https://arxiv.org/abs/1909.09436

    2020 arXiv

  2. [10]

    JetBrains. 2024. Structural search and replace. https://www.jetbrains.com/help/idea/structural-search-and-replace. html

    2024

  3. [11]

    Gregory Kamradt. 2023. Needle In A Haystack - Pressure Testing LLMs. https://github.com/gkamradt/LLMTest_ NeedleInAHaystack/blob/main/README.md

    2023

  4. [12]

    Julia Lawall. 2023. On the Origins of Coccinelle. In Eelco Visser Commemorative Symposium (EVCS 2023) (Open Access Series in Informatics (OASIcs), Vol. 109) , Ralf Lämmel, Peter D. Mosses, and Friedrich Steimann (Eds.). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstu...

    2023 doi

  5. [13]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings ...

    2020

  6. [14]

    Hongyu Li, Seohyun Kim, and Satish Chandra. 2019. Neural Code Search Evaluation Dataset. arXiv:1908.09804 [cs.SE] https://arxiv.org/abs/1908.09804

    2019 arXiv

  7. [15]

    Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang

  8. [16]

    Microsoft. 2024. Visual Studio Code - Code Navigation. https://code.visualstudio.com/docs/editor/editingevolved

    2024

  9. [17]

    Rajdeep Mukherjee, Omer Tripp, Ben Liblit, and Michael Wilson. 2022. Static Analysis for AWS Best Practices in Python Code. In 36th European Conference on Object-Oriented Programming (ECOOP 2022) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 222) , Karim Ali...

    2022 doi

  10. [18]

    Stolee, and Sebastian Elbaum

    Caitlin Sadowski, Kathryn T. Stolee, and Sebastian Elbaum. 2015. How developers search for code: a case study. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, ...

    2015

  11. [19]

    Inc. Semgrep. 2024. Semgrep | Homepage. https://semgrep.dev/

    2024

  12. [20]

    Susan Elliott Sim, Medha Umarji, Sukanya Ratanotayanon, and Cristina V. Lopes. 2011. How Well Do Search Engines Support Code Retrieval on the Web? ACM Transactions on Software Engineering and Methodology 21, 1 (Dec. 2011), 1–25. https://doi.org/10.1145/2063239.2063243

    2011

  13. [21]

    SourceGraph. 2024. SourceGraph - Code Search. https://sourcegraph.com/code-search

    2024

  14. [22]

    Jeffrey Svajlenko and Chanchal K. Roy. 2015. Evaluating clone detection tools with BigCloneBench. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) . 131–140. https://doi.org/10.1109/ICSM.2015. 7332459

    2015 doi

  15. [23]

    Inc. Tabnine. 2024. Tabnine AI Code Assistant. https://www.tabnine.com/

    2024

  16. [24]

    Rijnard van Tonder. 2024. Comby - Structural code search and replace for every language. https://comby.dev/

    2024

  17. [2024]

    arXiv:2406.06025 [cs.SE] https://arxiv.org/abs/2406

    RepoQA: Evaluating Long Context Code Understanding. arXiv:2406.06025 [cs.SE] https://arxiv.org/abs/2406. 06025

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.