Pith. sign in

REVIEW 3 major objections 6 minor 36 references

CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that retrieving parameter guidance from a similar prior design lets an LLM tune EDA flows to 9.9% lower power than standard optimizers.

desk verdict CROP is a genuinely new LLM+RAG system for EDA tuning with a plausible but under-supported central claim; the retrieval benefit is not yet isolated from 'any guidance helps.' read the letter →

arxiv 2507.02128 v2 pith:LDU4M5CJ submitted 2025-07-02 cs.LG

Jingyu Pan , Isaac Jacobson , Zheng Zhao , Tung-Chieh Chen , Guanglei Zhou , Chen-Chia Chang , Vineet Rashingkar , Yiran Chen This is my paper · ORCID
classification cs.LG
keywords LLM-guideddesignflowtuningretrieval-augmentedgenerationcircuitembeddingsVLSIparameteroptimizationRTLcodesummarizationEDAqualityofresultssimilarityretrievalchippower
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CROP is an automated EDA flow-tuning framework that replaces manual, expert-driven parameter selection with an LLM given prior tuning knowledge from similar circuits. The paper claims this is the first LLM-powered VLSI design flow tuner, and that its retrieval-augmented search converges faster and to better quality-of-results than random search, hyperparameter-optimization, and Bayesian-optimization baselines. On an industrial 5 nm processor whose retrieved similar design is a 7 nm predecessor, CROP reaches 127.08 mW after five iterations and 119.98 mW after 75, a 9.9% power reduction over the best baseline's 133.17 mW. Removing the retrieval component costs 6.59% at five iterations and 8.66% at 40 iterations, which isolates the contribution of prior-design guidance. If this generalizes, archived EDA runs become reusable knowledge instead of one-off experiments.

What carries the argument

Two-stage LLM-based design analysis is the load-bearing mechanism: the model first produces a structured summary for each RTL module, then merges those into a structured whole-design summary, and a text-embedding model turns the final summary into a dense vector. The database stores these vectors alongside the top-k parameter configurations and their measured QoR from earlier tuning runs. At query time, maximum inner product search (MIPS) selects the closest prior design, and its top-k guidance is inserted into the prompt of a chain-of-thought reasoning LLM that proposes one parameter vector per iteration; the previous proposals' measured QoR feed back into the prompt so the search is both constrained by history and informed by retrieved knowledge.

What would settle it

Run CROP on a target design while forcing retrieval to return the database design with the lowest embedding similarity; if the resulting search is no better than the same LLM search with retrieval disabled, the transfer premise is falsified. A complementary check would compute the Spearman correlation between a retrieved design's top-k parameter values and the target design's optimal settings across many design pairs; the correlation should approach zero when embeddings are dissimilar.

Watch

Extended reading notes

Core claim

The paper's central claim is that a chip design's RTL source can be reduced, via two levels of LLM summarization, to a dense embedding that captures enough of the circuit's function and structure to retrieve useful tuning experience from an archive. On a corpus of 96 circuits spanning 14 categories, this retrieval achieves an average precision of 0.906 at k=1, and on an industrial 5 nm processor core it identifies the 7 nm predecessor core as the nearest neighbor. Feeding that neighbor's top parameter configurations into a reasoning LLM's prompt lets the search reach 127.08 mW after only five iterations and 119.98 mW after 75, a 9.9% reduction over the best baseline result of 133.17 mW. The conclusion the authors draw is that semantically similar designs share the most important parameter sensitivities, and that an LLM constrained by retrieved guidance can emulate an experienced engineer's tuning practice.

Load-bearing premise

The load-bearing premise is that two chip designs whose LLM-written summaries have similar text embeddings will also have similar optimal EDA parameter settings, even when the designs use different technology nodes.

Editorial extensions

If this is right

  • Because CROP reaches a better setting in 5 iterations than the best baseline reaches in 75, early stopping after a handful of runs becomes a practical strategy for expensive EDA flows.
  • The 9.9% power reduction comes from transferring guidance between two architecturally similar cores at different technology nodes, so archived tuning results can stay useful as process nodes change.
  • The two-stage summarization lets the framework handle RTL codebases that exceed any single LLM context window, because only the structured summaries, not raw source, are embedded.
  • The RAG benefit numbers imply that the LLM alone, without retrieval, performs roughly on par with standard optimizers; the database, not the LLM, is what creates the decisive advantage.
  • CROP is model-agnostic and tool-agnostic, so it can be stacked with existing warm-start, early-stopping, and parallel trust-region search methods to reduce the dominant EDA-flow runtime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is to replace LLM-generated summary embeddings with embeddings of raw RTL, netlists, or simulation traces and compare which representation best predicts shared optimal parameters.
  • If the embedding-similarity premise is the real engine, then deliberately retrieving the least similar database design should make CROP's advantage over un-retrieved LLM search disappear; that experiment would separate retrieval quality from LLM search skill.
  • The retrieval precision numbers come from a 96-design open corpus, while the 9.9% improvement comes from a single industrial pair, so the production value depends on whether retrieval quality holds on larger, messier design databases.
  • The same retrieval-plus-prompt loop could transfer to other engineering settings with archived run logs but sparse written expertise, such as compiler flag tuning or analog circuit sizing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CROP, an LLM-based framework for automatically tuning EDA flow parameters. CROP first summarizes RTL source code into structured natural-language descriptions via an LLM, embeds those summaries with a text-embedding model, and stores the embeddings together with top-performing parameter configurations from previous optimization runs. For a new target design, CROP computes its embedding, retrieves the most similar design from the database using maximum inner product search, and injects that design's parameter guidance into the prompt of an LLM-based search engine. The LLM then proposes parameter configurations that are evaluated with Synopsys RTL Architect. Experiments on the RTLRewriter benchmark show high retrieval precision on module-level categories. On two proprietary industrial processor cores implemented in different technology nodes (Prior Core and Target Core), CROP finds lower power consumption than random search, Optuna, and Bayesian optimization, including a claimed 9.9% reduction in total power on Target Core. An ablation against a naive LLM search without retrieved guidance shows a 6.59% improvement after 5 iterations and 8.66% after 40 iterations.

Significance. If the reported results are robust, CROP would be a practically useful demonstration that retrieval-augmented LLM guidance can accelerate EDA parameter tuning and transfer knowledge across similar designs. The paper's strengths include a clearly described two-stage methodology, an ablation that isolates the RAG component from a naive LLM search, a runtime breakdown showing that the EDA flow itself dominates cost, and a parameter-importance analysis that attempts to justify cross-design transfer. However, the central claim that embedding-based similarity retrieval is the mechanism driving the QoR gains is not yet fully supported: the industrial database contains only one candidate prior design, so the retrieval step is effectively a fixed lookup, and the transfer hypothesis is not tested with dissimilar designs. Furthermore, the headline 9.9% result is based on a single optimization trace per method, with no variance or statistical testing. These are load-bearing issues for the paper's main contribution, but they are addressable within the scope of a revision.

major comments (3)
  1. [Section V-D] The RAG benefit analysis compares CROP against a naive LLM search without any retrieved guidance, which cannot distinguish 'retrieval of a semantically similar design helps' from 'including any concrete parameter guidance in the prompt helps.' Since the industrial database contains only one candidate prior design (Prior Core), the retrieval step is effectively a fixed lookup; the retrieval-precision experiments in Section V-B are computed on RTLRewriter module embeddings and do not establish that similarity-ranked retrieval improves QoR on the full-flow task. A control that retrieves a deliberately dissimilar design, or that injects non-similar/random parameter guidance, is needed to attribute the observed 6.59% and 8.66% improvements to similarity-based retrieval.
  2. [Section V-C, Table VI] The headline 9.9% power reduction over the best baseline rests on a single optimization trace for each method; no variance or statistical test is reported. Because random search and Bayesian optimization are stochastic, and because the LLM search itself may be sensitive to prompt or sampling randomness, the comparisons at 5, 10, 50, and 75 iterations could shift substantially across runs. Please report multiple independent runs (e.g., 5 or more seeds) with error bars and, where possible, a significance test or at least a clear statement of the number of runs used.
  3. [Section V-E] The parameter-importance analysis reveals notable differences between Prior Core and Target Core (e.g., power_weight_stdev 0.185 vs 0.018; enable_clock_latency_aware 0.018 vs 0.234; congestion_style 0.006 vs 0.161). This weakens the premise, stated in Section IV-C, that higher embedding similarity implies greater sharing of optimal parameter settings. The paper does not report the embedding similarity score between the two cores, nor does it compare the parameter-importance vectors of the two cores against those of a dissimilar design. Without such evidence, the knowledge-transfer mechanism underlying the retrieval component remains under-supported.
minor comments (6)
  1. [Title] The title contains a typo: 'O ptimization' should be 'Optimization'.
  2. [Abstract and Section I] The abstract states a '9.9% reduction in power consumption' without specifying the comparison basis, while Section I says 'compared to the best of the baselines.' Please state the comparison basis consistently in both places.
  3. [Section II-C and Section V-D] In Section II-C, 'these works highlight that there are significant benefits' is grammatically awkward; in Section V-D, 'tracking how they effect the result' should be 'affect.'
  4. [Section V-B] Table V reports Recall@1 of only 0.193, but the text emphasizes precision only; please also discuss the recall trade-off and its implications for retrieval-augmented search.
  5. [Section V-B] Precision@1 for the constant (0.250) and algebraic (0.000) categories is based on only 4 and 2 embeddings, respectively; please note these small sample sizes when interpreting the average precision.
  6. [Section V-F] The runtime breakdown is informative, but it would be helpful to state the wall-clock time per EDA flow iteration so that the amortization of RAG overhead can be quantified for larger budgets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CROP's 9.9% result is an empirical comparison against independent baselines; the retrieval premise is an untested transfer assumption, not a derivation that reduces to its inputs.

full rationale

The paper makes no mathematical derivation in which an output is defined in terms of the result it claims to predict. CROP's retrieval pipeline (Section IV) constructs design embeddings via LLM summarization and MIPS matching, and the parameter guidance is compiled from pre-existing optimization data (Section IV-A). The claimed 9.9% power reduction on Target Core (Section V-C, Table VI) compares CROP against independently implemented random search, Optuna, and Bayesian optimization baselines, none of which are fitted to CROP's outputs. The RAG benefit analysis (Section V-D) compares CROP against a naive LLM search without retrieval, giving an empirical control for the retrieval component, albeit not a dissimilar-design control. The load-bearing premise that embedding similarity predicts similar optimal parameter settings (Section IV-C) is an empirical assumption that may be under-tested: the retrieval-precision evaluation in Section V-B is on RTLRewriter modules, not on full-flow parameter transfer, and Section V-E shows some parameter importances differ between Prior Core and Target Core. These are soundness and selection-bias concerns, not circularity, because none of the compared quantities are defined in terms of the others and no load-bearing result is imported solely from a self-citation. The self-citation [14] is a survey and does not carry the argument. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by domain assumptions about transferability of tuning knowledge across designs, plus a few hand-chosen experimental constants. There are no free parameters in a fitted mathematical model and no invented physical entities.

free parameters (3)
  • top-k parameter guidance size k = not reported
    The number of best parameter sets stored as 'parameter guidance' affects prompt length and search behavior; Section IV-D never states k.
  • Spearman importance threshold |rho| > 0.10 = 0.10
    Used in Section V-E to define 'top-impact parameters'; the cut is arbitrary and not justified.
  • Set of 31 placement parameters and their value grids = 31 parameters, search space over 2.8e15 combinations
    The choice of which RTL Architect placement options to tune is an expert selection, not a derived quantity; the reported search-space size depends on it.
assumptions (5)
  • domain assumption Semantically similar designs, as measured by text embeddings of LLM summaries, tend to have similar optimal EDA parameter settings.
    Core transfer assumption behind retrieval-augmented search, invoked in Section IV-C and evaluated only on one similar core pair.
  • domain assumption The all-MiniLM-L6-v2 embeddings of natural-language design summaries capture circuit characteristics relevant to placement optimization.
    No validation beyond retrieval precision on RTLRewriter categories; retrieval precision does not establish QoR relevance.
  • domain assumption OpenAI o1 and GPT-4o provide parseable, valid parameter suggestions that improve over random search.
    LLM outputs may be invalid; a validation step is mentioned, but error rates are not reported.
  • domain assumption The baseline implementations of Optuna and Bayesian optimization are representative and correctly configured.
    Baseline details are minimal, with no hyperparameters, seeds, or code; defaults may not be optimal for this search space.
  • domain assumption Power is an appropriate single objective, and the PPA correlations justify optimizing power alone.
    Section V-G uses correlations from the same runs to justify the power focus; the timing trade-off is noted but not handled in the search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs." pith.science (2026). https://pith.science/paper/LDU4M5CJ

@misc{pith2026250702128,
  author       = {Pith},
  title        = {Pith review of: CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDU4M5CJ}},
  note         = {Machine review of arXiv:2507.02128}
}
read the original abstract

Modern very large-scale integration (VLSI) design requires the implementation of integrated circuits using electronic design automation (EDA) tools. Due to the complexity of EDA algorithms, the vast parameter space poses a huge challenge to chip design optimization, as the combination of even moderate numbers of parameters creates an enormous solution space to explore. Manual parameter selection remains industrial practice despite being excessively laborious and limited by expert experience. To address this issue, we present CROP, the first large language model (LLM)-powered automatic VLSI design flow tuning framework. Our approach includes: (1) a scalable methodology for transforming RTL source code into dense vector representations, (2) an embedding-based retrieval system for matching designs with semantically similar circuits, and (3) a retrieval-augmented generation (RAG)-enhanced LLM-guided parameter search system that constrains the search process with prior knowledge from similar designs. Experiment results demonstrate CROP's ability to achieve superior quality-of-results (QoR) with fewer iterations than existing approaches on industrial designs, including a 9.9% reduction in power consumption.

Figures

Figures reproduced from arXiv: 2507.02128 by the authors.

Figure 1
Figure 1. Overview of CROP. In stage 1, CROP constructs a design database. CROP first performs the two-step LLM-based design analysis [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Structure for LLM-based design analysis and embedding [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A t-SNE-based visualization of design description embeddings [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of CROP and naive LLM search (without RAG [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparing the total power usage found by CROP and the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The absolute Spearman’s rank correlation between parameters [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Correlation analysis between power and other QoR values for [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    Ptpt: Physical design tool parameter tuning via multi-objective bayesian optimization,

    H. Geng, T. Chen, Y . Ma, B. Zhu, and B. Yu, “Ptpt: Physical design tool parameter tuning via multi-objective bayesian optimization,” IEEE transactions on computer-aided design of integrated circuits and sys- tems, vol. 42, no. 1, pp. 178–189, 2022

    work page 2022

  2. [2]

    Fist: A feature-importance sampling and tree-based method for automatic design flow parameter tuning,

    Z. Xie, G.-Q. Fang, Y .-H. Huang, H. Ren, Y . Zhang, B. Khailany, S.-Y . Fang, J. Hu, Y . Chen, and E. C. Barboza, “Fist: A feature-importance sampling and tree-based method for automatic design flow parameter tuning,” in 2020 25th Asia and South Pacific Design Automation Con- ference (ASP-DAC). IEEE, 2020, pp. 19–25

    work page 2020

  3. [3]

    Ppatuner: pareto-driven tool parameter auto-tuning in physical design via gaussian process transfer learning,

    H. Geng, Q. Xu, T.-Y . Ho, and B. Yu, “Ppatuner: pareto-driven tool parameter auto-tuning in physical design via gaussian process transfer learning,” in Proceedings of the 59th ACM/IEEE Design Automation Conference, ser. DAC ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1237–1242. [Online]. Available: https://doi.org/10.1145/34895...

    arXiv 2022

  4. [4]

    Boosting vlsi design flow parameter tuning with random embedding and multi- objective trust-region bayesian optimization,

    S. Zheng, H. Geng, C. Bai, B. Yu, and M. D. Wong, “Boosting vlsi design flow parameter tuning with random embedding and multi- objective trust-region bayesian optimization,” ACM Transactions on Design Automation of Electronic Systems, vol. 28, no. 5, pp. 1–23, 2023

    work page 2023

  5. [5]

    A synthesis-parameter tuning system for autonomous design-space exploration,

    M. M. Ziegler, H.-Y . Liu, G. Gristede, B. Owens, R. Nigaglioni, and L. P. Carloni, “A synthesis-parameter tuning system for autonomous design-space exploration,” in 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE) , 2016, pp. 1148–1151

    work page 2016

  6. [6]

    Cad tool design space exploration via bayesian optimization,

    Y . Ma, Z. Yu, and B. Yu, “Cad tool design space exploration via bayesian optimization,” in 2019 ACM/IEEE 1st Workshop on Machine Learning for CAD (MLCAD) . IEEE, 2019, pp. 1–6

    work page 2019

  7. [7]

    Optuna: A next- generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next- generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , 2019, pp. 2623–2631

    2019

  8. [8]

    A learning-based rec- ommender system for autotuning design flows of industrial high- performance processors,

    J. Kwon, M. M. Ziegler, and L. P. Carloni, “A learning-based rec- ommender system for autotuning design flows of industrial high- performance processors,” in Proceedings of the 56th Annual Design Automation Conference 2019 , 2019, pp. 1–6

    work page 2019

Show all 36 references
  1. [9]

    Vlsi placement parameter optimization using deep reinforcement learning,

    A. Agnesina, K. Chang, and S. K. Lim, “Vlsi placement parameter optimization using deep reinforcement learning,” in Proceedings of the 39th international conference on computer-aided design , 2020, pp. 1–9

    2020

  2. [10]

    Flowtuner: A multi-stage eda flow tuner exploiting parameter knowledge transfer,

    R. Liang, J. Jung, H. Xiang, L. Reddy, A. Lvov, J. Hu, and G.-J. Nam, “Flowtuner: A multi-stage eda flow tuner exploiting parameter knowledge transfer,” in 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD) , 2021, pp. 1–9

    2021

  3. [11]

    Fasttuner: Transferable physical design parameter optimization using fast reinforce- ment learning,

    H.-H. Hsiao, Y .-C. Lu, P. Vanna-Iampikul, and S. K. Lim, “Fasttuner: Transferable physical design parameter optimization using fast reinforce- ment learning,” in Proceedings of the 2024 International Symposium on Physical Design, 2024, pp. 93–101

    2024

  4. [12]

    Ranktuner: When design tool parameter tuning meets preference bayesian optimization,

    P. Xu, S. Zheng, Y . Ye, C. Bai, S. Xu, H. Geng, T.-Y . Ho, and B. Yu, “Ranktuner: When design tool parameter tuning meets preference bayesian optimization,” in 2024 IEEE/ACM International Conference On Computer Aided Design (ICCAD) , 2024

    2024

  5. [13]

    Nautilus: Fast automated ip design space search using guided genetic algorithms,

    M. K. Papamichael, P. Milder, and J. C. Hoe, “Nautilus: Fast automated ip design space search using guided genetic algorithms,” in Proceedings of the 52nd Annual Design Automation Conference , 2015, pp. 1–6

    2015

  6. [14]

    A survey of research in large language models for electronic design automation,

    J. Pan, G. Zhou, C.-C. Chang, I. Jacobson, J. Hu, and Y . Chen, “A survey of research in large language models for electronic design automation,” ACM Transactions on Design Automation of Electronic Systems , 2025

    2025

  7. [15]

    Rtlrewriter: Methodologies for large models aided rtl code optimization,

    X. Yao, Y . Wang, X. Li, Y . Lian, R. Chen, L. Chen, M. Yuan, H. Xu, and B. Yu, “Rtlrewriter: Methodologies for large models aided rtl code optimization,” in Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design , 2024, pp. 1–7

    2024

  8. [16]

    The dawn of ai-native eda: Op- portunities and challenges of large circuit models,

    L. Chen, Y . Chen, Z. Chu, W. Fang, T.-Y . Ho, R. Huang, Y . Huang, S. Khan, M. Li, X. Li et al. , “The dawn of ai-native eda: Op- portunities and challenges of large circuit models,” arXiv preprint arXiv:2403.07257, 2024

    2024 arXiv

  9. [17]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates ...

    2017

  10. [18]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Ch...

    2020

  11. [19]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  12. [20]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22, 2022

    2022

  13. [21]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” 2023. [Online]. Available: https: //arxiv.org/abs/2305.06599

    2023 arXiv

  14. [22]

    Learning to reason with llms,

    OpenAI, “Learning to reason with llms,” 2024. [Online]. Available: https://openai.com/index/learning-to-reason-with-llms/

    2024

  15. [23]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in Proceedings of the 34th International Conference on Neural Inf...

    2020

  16. [24]

    KILT: a benchmark for knowledge intensive language tasks,

    F. Petroni, A. Piktus, A. Fan, P. S. H. Lewis, M. Yazdani, N. D. Cao, J. Thorne, Y . Jernite, V . Plachouras, T. Rockt ¨aschel, and S. Riedel, “KILT: a benchmark for knowledge intensive language tasks,” CoRR, vol. abs/2009.02252, 2020

    2009 arXiv

  17. [25]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. Oguz, S. Min, L. Wu, S. Edunov, D. Chen, and W. Yih, “Dense passage retrieval for open-domain question answering,” CoRR, vol. abs/2004.04906, 2020

    2004 arXiv

  18. [26]

    A fast parameter tuning framework via transfer learning and multi-objective bayesian optimization,

    Z. Zhang, T. Chen, J. Huang, and M. Zhang, “A fast parameter tuning framework via transfer learning and multi-objective bayesian optimization,” in Proceedings of the 59th ACM/IEEE Design Automation Conference, ser. DAC ’22. New York, NY , USA: Association for Computing Machine...

    2022

  19. [27]

    CURIE: Evaluating LLMs on multitask scientific long-context understanding and reasoning,

    H. Cui, Z. Shamsi, G. Cheon, X. Ma, S. Li, M. Tikhanovskaya, P. C. Norgaard, N. Mudur, M. B. Plomecka, P. Raccuglia, Y . Bahri, V . V . Albert, P. Srinivasan, H. Pan, P. Faist, B. A. Rohr, M. J. Statt, D. Morris, D. Purves, E. Kleeman, R. Alcantara, M. Abraham, M. Mohammad, E....

    2025

  20. [28]

    Dealing with cat- egorical and integer-valued variables in bayesian optimization with gaussian processes,

    E. C. Garrido-Merch ´an and D. Hern ´andez-Lobato, “Dealing with cat- egorical and integer-valued variables in bayesian optimization with gaussian processes,” Neurocomputing, vol. 380, pp. 20–35, 2020

    2020

  21. [29]

    Bayesian Optimization: Open source constrained global optimization tool for Python,

    F. Nogueira, “Bayesian Optimization: Open source constrained global optimization tool for Python,” 2014–. [Online]. Available: https://github.com/bayesian-optimization/BayesianOptimization

    2014

  22. [30]

    The proof and measurement of association between two things

    C. Spearman, “The proof and measurement of association between two things.” 1961

    1961

  23. [31]

    Loogle: Can long- context language models understand long contexts?

    J. Li, M. Wang, Z. Zheng, and M. Zhang, “Loogle: Can long- context language models understand long contexts?” arXiv preprint arXiv:2311.04939, 2023

    2023 arXiv

  24. [32]

    A systematic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN international symposium on machine programming , 2022, pp. 1–10

    2022

  25. [33]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

    2024

  26. [34]

    Sled: Self logits evolution decoding for improving factuality in large language models,

    J. Zhang, D.-C. Juan, C. Rashtchian, C.-S. Ferng, H. Jiang, and Y . Chen, “Sled: Self logits evolution decoding for improving factuality in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2411.02433

    2025 arXiv

  27. [35]

    Contrastive decoding improves reasoning in large language models,

    S. O’Brien and M. Lewis, “Contrastive decoding improves reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/ 2309.09117

    2023 arXiv

  28. [1901]

    Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.