Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Endpoint-by-endpoint chunking beats naive splitting for RAG-based API discovery

desk verdict Useful benchmark and chunking comparison, but the significance analysis contradicts itself and the LLM-generated ground truth makes the agent gains hard to trust. read the letter →

arxiv 2505.19310 v1 pith:NMX24IMW submitted 2025-05-25 cs.SE cs.AI

Robin D. Pesl , Jerin G. Mathew , Massimo Mecella , Marco Aiello This is my paper · ORCID
classification cs.SEcs.AI
keywords servicediscoveryretrieval-augmentedgenerationOpenAPIchunkingstrategiesLLMagentsendpointSOCBench-DRestBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation can shrink the enormous documentation of an OpenAPI service registry down to the few endpoints an LLM needs, but the best way to cut that documentation into retrievable pieces was unknown. The paper argues that splitting each OpenAPI file per endpoint—and, for the best accuracy, embedding a short LLM-written summary of each endpoint while keeping the full endpoint details as the retrieved chunk—reliably beats naive whole-document and JSON-tree chunking. On its new 550-query benchmark spanning eleven industry domains, the retrieval count $k$ and the choice of embedding model affected accuracy more than any chunking refinement, and no endpoint-based strategy was significantly better than another across domains. The paper further claims that a "Discovery Agent"—an LLM that decomposes the user query, retrieves endpoint summaries, then fetches full details on demand—raises precision and cuts token count, but lowers recall because it filters out relevant endpoints too aggressively. If these findings hold, the practical recipe is simple: split by endpoint, use a strong embedding model, pick the largest $k$ the prompt budget allows, and treat an agent as a precision-and-cost feature rather than a recall feature.

What carries the argument

The central machinery is the pairing of a splitting method with a refinement step. The OpenAPI file is split either not at all, into JSON leaf nodes, or into individual endpoints; the refinement then either re-chunks tokens, strips examples, keeps only a few representative fields, or has an LLM generate a query or a short summary per endpoint for embedding. The load-bearing variant keeps the full endpoint as the chunk content but embeds the LLM's endpoint summary, so query-to-endpoint matching happens in summary space while the prompt receives full details. A second component, the Discovery Agent, exposes two tools to the LLM—one that retrieves top-$k$ endpoint summaries, one that returns a single endpoint's full specification on demand—so the model can split the query, filter candidates, and fetch details only for endpoints it deems indispensable. The evaluation machinery is SOCBench-D, which generates five services of ten endpoints per industry domain, writes queries over random endpoint subsets, and validates both services and query-to-endpoint solutions with LLM checks plus a formal OpenAPI validator.

What would settle it

Run the same chunking and retrieval pipelines on a version of SOCBench-D (or RestBench) whose expected-endpoint sets have been re-checked by human annotators, and recompute recall and precision; if whole-document or JSON chunking draws even with endpoint-split chunking on the human-relevant ground truth, the paper's central ranking is an artifact of LLM-generated labels. A cheaper check: take a sample of the 550 SOCBench-D queries and have two annotators flag endpoints that the query could be satisfied without, then see whether the endpoint-split advantage holds when those flagged endpoints are removed from the expected set.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a preprocessing recipe for retrieval-augmented endpoint discovery and a benchmark for measuring it. It claims that endpoint-based chunking strategies outperform whole-document and JSON-tree chunking on both its synthetic cross-domain benchmark and a real-world benchmark of two widely used APIs; among endpoint-based strategies, differences are not statistically significant across domains, so the authors recommend the simplest implementation. The dominant accuracy factors are the retrieval size $k$ and the embedding model, with a large open-source model leading, a proprietary model close behind, and the smallest model clearly worst. For further token reduction, the Discovery Agent is claimed to improve precision substantially while being prone to decrease recall; even when given all endpoints, the agent leaves relevant ones out, exposing where LLM reasoning, not retrieval, is the bottleneck. The paper also contributes SOCBench-D, five benchmark instances of fifty-five services and 550 natural-language-to-endpoint queries across the eleven GICS industry sectors, created and validated with LLMs.

Load-bearing premise

The benchmark's "correct" answers—which endpoints a query requires—are written and validated by LLMs, so the measured superiority of the recommended chunking strategy could be an artifact of the LLM's own sense of relevance rather than what a human integrator or downstream system would actually need.

Editorial extensions

If this is right

  • A service-composition pipeline that feeds an LLM only the top retrieved endpoints can keep prompts inside context limits while preserving the information needed to plan a composition; endpoint-split chunking is the safe preprocessing default.
  • Raising $k$ is the most direct way to trade tokens for recall: the paper's Pareto analysis always shows recall climbing and precision falling as $k$ goes from 5 to 10 to 20.
  • Choosing the embedding model is the second lever; the small open-source model consistently lagged, and the authors recommend picking the strongest model the hardware or budget allows.
  • The Discovery Agent reduces the final prompt to a small set of verified endpoints, so it is a viable tool for lowering cost and keeping noise out of the prompt; its recall loss means it should not be used where missing an endpoint is expensive.
  • SOCBench-D gives a fixed, cross-domain target for future work on endpoint retrieval, with the caveat that its ground truth was produced by the same kind of LLM that the retrieval system uses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the summary-embedding result generalizes, then for other API-consuming LLM tasks (test generation, documentation drift detection, schema extraction) the same "embed a distilled view, retrieve full detail" split should be the first thing to try.
  • The agent's recall loss suggests a concrete fix the authors did not test: add a second-pass check in which the agent is forced to justify each kept endpoint against the original query, or lower the filtering threshold when the downstream task tolerates noise.
  • Because the ground truth is LLM-generated, the paper's rankings measure agreement with an LLM's notion of relevance; a human-annotated subset of SOCBench-D queries would show whether the endpoint-split advantage survives contact with human expectations.
  • Token accounting in the paper counts the prompt the LLM sees, but the agent pays an extra LLM call per query; an extension that measures end-to-end cost (retrieval plus agent calls) would clarify when the precision gain is actually worth the latency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies retrieval-augmented generation (RAG) for endpoint discovery from OpenAPI service descriptions. It proposes a new synthetic benchmark, SOCBench-D, built from GICS domains, and compares nine chunking strategies (token-based and LLM-based) across three embedding models and three top-k values, evaluating recall and precision on SOCBench-D and RestBench. It also introduces a Discovery Agent that retrieves endpoint summaries and fetches full endpoint details on demand. The main claims are that endpoint-based chunking, especially LLM-generated summaries, outperforms whole-document and JSON-split chunking; that the embedding model and k are the dominant factors; and that the Discovery Agent improves precision at the cost of recall.

Significance. If the findings hold, the paper provides actionable guidance for preprocessing OpenAPI specifications for RAG-based service discovery, a practically relevant problem given LLM token limits and API documentation scale. The SOCBench-D benchmark is a potentially reusable resource with 550 queries across 11 domains, and the authors release code and data. The study is broad in its coverage of chunking strategies, embedding models, and k values, and the comparison against RestBench adds a real-world check. However, the validity of the central comparisons depends on the independence of the benchmark ground truth from the evaluated methods, and on the consistency of the statistical reporting; both issues currently require additional work before the claims can be accepted.

major comments (3)
  1. [Section IV-B, Algorithms 1 and 2, and Section IV-F] The SOCBench-D ground truth is generated, revised, and validated by OpenAI LLMs, and the best-performing summary chunking strategy and the Discovery Agent also rely on OpenAI LLMs. Specifically, createQuery in Algorithm 2 asks the LLM to write a query for a selected endpoint set, asks the same LLM family to list further endpoints, and then iteratively rewrites the query until checkNecessary returns exactly the expected endpoints. This creates a closed evaluation loop: the benchmark measures how well retrieval aligns with the generating LLM's own notion of endpoint necessity, rather than how well retrieval serves realistic human-written queries. Because every comparison in Section IV-D and the Discovery Agent evaluation in Section IV-F uses this ground truth, the concern is load-bearing. The RestBench validation is too narrow to resolve it, as the paper itself notes in Section IV-G that RestBench covers only Spotify and TMDB, i.e., two services in one domain. I request either a held-out set of human-written queries over the same OpenAPIs, or a version of SOCBench-D generated with a different model family, to show that the ranking of chunking strategies and the agent advantage are not artifacts of the generating LLM's semantic preferences.
  2. [Section IV-D, Figure 6a, and the accompanying text] The Friedman test results in Figure 6a report p=0.00 for the All row for every model and every k value. Under the stated 5% significance level, p=0.00 means the test rejects the null hypothesis that the chunking strategies perform equally across all domains. The text immediately after the figure states that 'we can assume that there is no significant difference between chunking strategies over all domains,' which is the opposite of the reported statistic. This contradiction is load-bearing because the recommendation to choose the simplest endpoint-split strategy rests on the claim of no significant difference. Please correct either the statistical test, its interpretation, or the table entries, and clarify whether values below or above 0.05 are marked in bold, since the current wording says 'entries exceeding the significance level' but p=0.00 is not bold.
  3. [Abstract and Section IV-F, Figure 10] The abstract states that 'Relying on an agent significantly improves precision,' but Figure 10 presents the Discovery Agent comparison only as a scatterplot of point estimates, without confidence intervals, error bars, or a significance test. The paper applies the Friedman test to chunking strategies but does not report any statistical test for the agent-versus-summary comparison. The word 'significantly' currently has no statistical backing. Please add an appropriate significance test, or qualify the claim as an observed improvement in precision rather than a statistically significant one.
minor comments (5)
  1. [Abstract] There are typographical errors in the abstract: 'oken' should be 'token' in 'input oken limitations' and 'nd' should be 'and' in 'nd retrieves specification details on demand.'
  2. [Section IV-B, after Algorithm 2] The paragraph beginning 'To create the queries, we rely on createQueries...' is duplicated verbatim; one copy should be removed.
  3. [Figure 6b, caption] The caption says the token count is 'Averaged over k=5,10,20,' but the token count per chunk is independent of k, since k is a retrieval parameter and does not affect chunk construction. Please clarify what is being averaged or remove the reference to k.
  4. [Figure 6a, caption and text] The caption says 'Entries exceeding the significance level are marked in bold,' but for a significance test at the 5% level, the relevant entries are those below 0.05. Please reword to 'Entries below the significance level are marked in bold' or otherwise make the convention unambiguous.
  5. [Table I and Section III-B] The CRAFT implementation is described as 'adapted to exactly return k results' and to use summary, endpoint name, and endpoint description views, but the original CRAFT method is a multi-view retrieval approach; please clarify in the text that this is a simplified adaptation rather than the original algorithm.

Circularity Check

1 steps flagged · score 4.0 of 10

SOCBench-D ground truth is iteratively rewritten until the generating LLM's own necessity checks return the chosen endpoints; the best chunking strategy and Discovery Agent are built on the same LLM family, creating a partially closed evaluation loop.

  1. other [Section IV-B, Algorithm 2 (createQuery and checkNecessary, lines 52-57); Section III-B (Summary strategy); Section IV-A (Discovery Agent tools)]
    "If there is a mismatch between e_necessary and e_expected, we prompt the LLM in a chat-based manner (57), i.e., in a question-answer style, with the expected endpoints e_expected, the additional endpoints e_necessary \ e_expected, and the absent endpoints e_expected \ e_necessary in the response message to the LLM to improve the prompt and continue with (54)."

    Algorithm 2 constructs each SOCBench-D query by sampling a random expected endpoint set, asking an LLM to write a query for it, asking the LLM to list further endpoints, and then rewriting the query until checkNecessary, again an LLM, returns exactly the sampled set. The benchmark's target endpoints are therefore fitted to the generating LLM's notion of necessity rather than sampled from real user intent. The strongest reported chunking strategy (Summary) and the Discovery Agent generate their retrieval representations with the same LLM family: an LLM produces endpoint summaries, and the agent is an OpenAI LLM. Consequently, the measured recall and precision partly measure agreement between the retriever and the test-set generator rather than independent endpoint-discovery correctness.

full rationale

The paper's central empirical claims rest on SOCBench-D, whose query-endpoint ground truth is generated and iteratively validated by an OpenAI LLM. Algorithm 2 rewrites each query until the same LLM family's necessity judgments match a randomly chosen expected endpoint set. Since the best-performing summary chunking strategy and the Discovery Agent also rely on LLM-generated summaries and an OpenAI agent, the evaluation partially measures how well the system aligns with the generating LLM's relevance judgments, which is a structural circularity in the benchmark. The paper's own discussion acknowledges that for SOCBench-D 'the query precisely aligns with the expected endpoints through an iterative process,' unlike RestBench. The RestBench evaluation provides genuinely external evidence, and the paper reports that its main findings are reinforced there, which prevents a score above 4; however, RestBench is limited to two services in one domain, so the concern is not fully resolved. No load-bearing self-citation, uniqueness-import, or ansatz-smuggling-through-citation step was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claims rest on the validity of the LLM-generated benchmark and the statistical interpretation of retrieval results. The main hand-chosen parameters are k, chunk sizes, query-size distribution, and similarity threshold. No new physical or mathematical entities are introduced beyond the benchmark and agent artifacts.

free parameters (4)
  • k (top-k retrieval) = {5, 10, 20}
    The number of chunks returned per query. The paper finds k is the biggest influence on recall and precision. It is hand-chosen as multiples of the benchmark's mean expected endpoint count.
  • chunk size s and overlap l = s={100, 200}, l={0, 20}
    Token chunking parameters selected for the LLM-generated OpenAPIs, which are shorter than real-world specifications. The authors note real-world OpenAPIs may require different settings.
  • SOCBench-D query size distribution = mean mu=5, sigma=2
    The cardinality of the expected endpoint set per query is drawn from a normal distribution with these hand-picked parameters, about 10% of endpoints within a domain.
  • query similarity threshold = 0.8
    Queries are discarded if their embedding similarity to existing queries in the domain exceeds this threshold, shaping the benchmark's ambiguity level.
assumptions (4)
  • domain assumption GICS sectors cover all relevant service domains, so SOCBench-D generalizes across domains
    Section IV-B states that because GICS is designed to encompass all industry sectors, further domains are subdomains, justifying generalizability across domains.
  • domain assumption OpenAPI is the state-of-practice for service descriptions
    Section I asserts OpenAPI is the de facto standard and bases the RAG system solely on it.
  • domain assumption LLM-generated OpenAPIs and queries are representative of real-world service documentation
    Section IV-G acknowledges generated services may be less extensive than real ones, but the benchmark still serves as a proxy for evaluation.
  • ad hoc to paper The LLM-based checkNecessary step correctly determines which endpoints are genuinely required by a query
    Algorithm 2 uses an LLM in a question-answer style to filter endpoints, and this LLM judgment defines the ground truth e_necessary.
invented entities (2)
  • SOCBench-D independent evidence
    purpose: A multi-domain synthetic benchmark of OpenAPI services, natural language queries, and expected endpoints for evaluating RAG endpoint discovery
    The benchmark is publicly released with data, allowing others to run it and falsify its properties.
  • Discovery Agent independent evidence
    purpose: An LLM agent that decomposes a query into tasks, retrieves endpoint summaries via RAG, and fetches details on demand to reduce token count and improve precision
    Prototype code is released on GitHub, though prompts and exact model versions are not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking." pith.science (2026). https://pith.science/paper/NMX24IMW

@misc{pith2026250519310,
  author       = {Pith},
  title        = {Pith review of: Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMX24IMW}},
  note         = {Machine review of arXiv:2505.19310}
}
read the original abstract

Integrating multiple (sub-)systems is essential to create advanced Information Systems. Difficulties mainly arise when integrating dynamic environments, e.g., the integration at design time of not yet existing services. This has been traditionally addressed using a registry that provides the API documentation of the endpoints. Large Language Models have shown to be capable of automatically creating system integrations (e.g., as service composition) based on this documentation but require concise input due to input oken limitations, especially regarding comprehensive API descriptions. Currently, it is unknown how best to preprocess these API descriptions. In the present work, we (i) analyze the usage of Retrieval Augmented Generation for endpoint discovery and the chunking, i.e., preprocessing, of state-of-practice OpenAPIs to reduce the input oken length while preserving the most relevant information. To further reduce the input token length for the composition prompt and improve endpoint retrieval, we propose (ii) a Discovery Agent that only receives a summary of the most relevant endpoints nd retrieves specification details on demand. We evaluate RAG for endpoint discovery using (iii) a proposed novel service discovery benchmark SOCBench-D representing a general setting across numerous domains and the real-world RestBench enchmark, first, for the different chunking possibilities and parameters measuring the endpoint retrieval accuracy. Then, we assess the Discovery Agent using the same test data set. The prototype shows how to successfully employ RAG for endpoint discovery to reduce the token count. Our experiments show that endpoint-based approaches outperform naive chunking methods for preprocessing. Relying on an agent significantly improves precision while being prone to decrease recall, disclosing the need for further reasoning capabilities.

Figures

Figures reproduced from arXiv: 2505.19310 by the authors.

Figure 1
Figure 1. RAG for Endpoint Discovery database using e. The chunk database compares e using a similarity metric with the embeddings of the service chunks contained in the database. The results are the top k most similar chunks according to the metric, which are then returned to the chunk retriever in 3 . Finally, in 4 , the chunk retriever forwards the retrieved results to the user, who can add them to their prompt either manu… view at source ↗
Figure 2
Figure 2. Overview of the Discovery Agent Approach for Endpoint Discovery [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Cross-Domain Average Analysis recall and precision as the performance criteria metrics be￾cause we are interested in how many correct endpoints we retrieve. We are not interested in the ranking of the endpoints because we assume that the incorrect endpoints are filtered out in a later stage, i.e., we do not consider other metrics like Mean Reciprocal Rank, which weigh positioning and Hit@k, which does not consider t… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Recall by Chunking Strategy as Boxplots Grouped by Model for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Statistical Stability Analysis of the Candidates. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Friedman Test for the Endpoint Split and Token Count per Chunk for All Chunking Strategies from Table [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Evaluation of Chunking Strategies with Non-Endpoint Split Splitting, i.e., Whole Document and JSON. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: RestBench Pareto Front Analysis of Recall and Precision as [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: RestBench Evaluation of Chunking Strategies with Non-Endpoint Split Splitting, i.e., Whole Document and JSON. Formatting as in Figure [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Pareto Front Analysis of the Discovery Agent as a Scatterplot. Agent Results are in Blue. The Summary Chunking Strategy is in Orange. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 31 canonical work pages

  1. [1]

    Verfahren zur Nutzung von un- bekannten neuen Systemdiensten in einer Fahrzeuganwendung,

    R. D. Pesl, K. Klein, and M. Aiello, “Verfahren zur Nutzung von un- bekannten neuen Systemdiensten in einer Fahrzeuganwendung,” German Patent DE 10 2024 108 126 A1, 2024

    work page 2024

  2. [2]

    Uncovering LLMs for service-composition: Challenges and opportunities,

    R. D. Pesl, M. Stötzner, I. Georgievski, and M. Aiello, “Uncovering LLMs for service-composition: Challenges and opportunities,” inICSOC 2023 WS. Springer, 2024

    work page 2023

  3. [3]

    Compositio Prompto: An architecture to employ large language models in automated service computing,

    R. D. Peslet al., “Compositio Prompto: An architecture to employ large language models in automated service computing,” inService-Oriented Computing. Springer Nature Singapore, 2025, pp. 276–286

    work page 2025

  4. [4]

    GPT-4 Turbo in the OpenAI API,

    OpenAI, “GPT-4 Turbo in the OpenAI API,” https://help.openai.com/ en/articles/8555510-gpt-4-turbo-in-the-openai-api, 2024, last accessed 2025-03-07

    arXiv 2024

  5. [5]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewiset al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” inNeurIPS, vol. 33. Curran Associates, 2020, pp. 9459– 9474

    work page 2020

  6. [6]

    Advanced system integration: Analyzing OpenAPI chunking for retrieval- augmented generation,

    R. D. Pesl, J. G. Mathew, M. Mecella, and M. Aiello, “Advanced system integration: Analyzing OpenAPI chunking for retrieval- augmented generation,” inCAiSE 2025. Springer Nature, 2025, to appear. [Online]. Available: https://arxiv.org/abs/2411.19804

    arXiv 2025

  7. [7]

    Global industry classification standard (GICS),

    MSCI Inc. and Standard & Poor’s, “Global industry classification standard (GICS),” https://www.msci.com/gics, August 2024

    work page 2024

  8. [8]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs,

    Y . Qinet al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” 2023

    work page 2023

Show all 46 references
  1. [9]

    RestGPT: Connecting large language models with real-world applications via restful APIs,

    Y . Songet al., “RestGPT: Connecting large language models with real-world applications via restful APIs,” 2023. [Online]. Available: https://arxiv.org/abs/2306.06624

    2023 arXiv

  2. [10]

    New embedding models and API updates,

    OpenAI, “New embedding models and API updates,” Jan 2024, last accessed 2024-07-18. [Online]. Available: https://openai.com/blog/ new-embedding-models-and-api-updates 14

    2024

  3. [11]

    Web service composition: A survey of techniques and tools,

    A. L. Lemos, F. Daniel, and B. Benatallah, “Web service composition: A survey of techniques and tools,”ACM Comput. Surv., vol. 48, no. 3, dec 2015

    2015

  4. [12]

    UPnP service discovery for heterogeneous networks,

    J. M. S. Santana, M. Petrova, and P. Mahonen, “UPnP service discovery for heterogeneous networks,” inIEEE PIMRC, vol. 17. IEEE, 2006, pp. 1–5

    2006

  5. [13]

    Unraveling the web services web: an introduction to SOAP, WSDL, and UDDI,

    F. Curberaet al., “Unraveling the web services web: an introduction to SOAP, WSDL, and UDDI,”IEEE Internet Computing, vol. 6, no. 2, pp. 86–93, 2002

    2002

  6. [14]

    A distributed approach for the federation of heterogeneous registries,

    L. Baresi and M. Miraz, “A distributed approach for the federation of heterogeneous registries,” inICSOC 2006. Springer, 2006, pp. 240–251

    2006

  7. [15]

    Dynamic device and service discovery extensions for ws-bpel,

    H. Bohn, F. Golatowski, and D. Timmermann, “Dynamic device and service discovery extensions for ws-bpel,” inICSSSM 2008. IEEE, 2008, pp. 1–6

    2008

  8. [16]

    Service discovery and orchestration for distributed service repositories,

    I. Fikouras and E. Freiter, “Service discovery and orchestration for distributed service repositories,” inICSOC 2003. Springer, 2003, pp. 59–74

    2003

  9. [17]

    Discovery of RESTful Web services based on the OpenAPI 3.0 standard with semantic annotations,

    A. T. Soki and F. Siqueira, “Discovery of RESTful Web services based on the OpenAPI 3.0 standard with semantic annotations,” inAINA. Springer, 2024, pp. 22–34

    2024

  10. [18]

    Microservices,

    J. Thönes, “Microservices,”IEEE software, vol. 32, no. 1, pp. 116–116, 2015

    2015

  11. [19]

    GPT-4 technical report,

    J. Achiamet al., “GPT-4 technical report,” 2023. [Online]. Available: https://arxiv.org/abs/2303.08774

    2023 arXiv

  12. [20]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  13. [21]

    Leveraging large language models to improve rest api testing,

    M. Kim, T. Stennett, D. Shah, S. Sinha, and A. Orso, “Leveraging large language models to improve rest api testing,” inICSE, vol. 44, 2024, pp. 37–41

    2024

  14. [22]

    Better language models and their implications,

    A. Radford, J. Wu, D. Amodei, D. Amodei, J. Clark, M. Brundage, and I. Sutskever, “Better language models and their implications,”OpenAI blog, vol. 1, no. 2, 2019

    2019

  15. [23]

    Attention is all you need,

    A. Vaswaniet al., “Attention is all you need,”NeurIPS, vol. 30, 2017

    2017

  16. [24]

    Improving language understanding by generative pre-training,

    A. Radfordet al., “Improving language understanding by generative pre-training,” 2018

    2018

  17. [25]

    Large language models for software engineering: Survey and open problems,

    A. Fanet al., “Large language models for software engineering: Survey and open problems,” 2023. [Online]. Available: https: //arxiv.org/abs/2310.03533

    2023 arXiv

  18. [26]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inNAACL-HLT 2019, 2019, pp. 4171–4186

    2019

  19. [27]

    The power of noise: Redefining retrieval for RAG systems,

    F. Cuconasuet al., “The power of noise: Redefining retrieval for RAG systems,” inSIGIR, vol. 47, 2024, pp. 719–729

    2024

  20. [28]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Weiet al., “Chain-of-thought prompting elicits reasoning in large language models,”NeurIPS, vol. 35, pp. 24 824–24 837, 2022

    2022

  21. [29]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yaoet al., “Tree of thoughts: Deliberate problem solving with large language models,”NeurIPS, vol. 36, 2024

    2024

  22. [30]

    WebGPT: Browser-assisted question-answering with human feedback,

    R. Nakanoet al., “WebGPT: Browser-assisted question-answering with human feedback,” 2021. [Online]. Available: https://arxiv.org/abs/2112. 09332

    2021

  23. [31]

    Training verifiers to solve math word problems,

    K. Cobbeet al., “Training verifiers to solve math word problems,”

  24. [32]

    Pal: Program-aided language models,

    L. Gaoet al., “Pal: Program-aided language models,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 10 764–10 799

    2023

  25. [33]

    Augmented language models: a survey,

    G. Mialonet al., “Augmented language models: a survey,” 2023. [Online]. Available: https://arxiv.org/abs/2302.07842

    2023 arXiv

  26. [34]

    Function calling and other API updates,

    OpenAI, “Function calling and other API updates,” Jun 2024, last accessed 2024-07-18. [Online]. Available: https://openai.com/index/ function-calling-and-other-api-updates/

    2024

  27. [35]

    React: Synergizing reasoning and acting in language models,

    S. Yaoet al., “React: Synergizing reasoning and acting in language models,” 2023. [Online]. Available: https://arxiv.org/abs/2210.03629

    2023 arXiv

  28. [36]

    API-Bank: A comprehensive benchmark for tool- augmented LLMs,

    M. Liet al., “API-Bank: A comprehensive benchmark for tool- augmented LLMs,” inEMNLP. Association for Computational Lin- guistics, 2023

    2023

  29. [37]

    Chain of tools: Large language model is an automatic multi-tool learner,

    Z. Shiet al., “Chain of tools: Large language model is an automatic multi-tool learner,” 2024. [Online]. Available: https://arxiv.org/abs/2405. 16533

    2024

  30. [38]

    CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets,

    L. Yuan, Y . Chen, X. Wang, Y . R. Fung, H. Peng, and H. Ji, “CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets,” 2024. [Online]. Available: https://arxiv.org/abs/2309.17428

    2024 arXiv

  31. [39]

    Talm: Tool augmented language models,

    A. Parisi, Y . Zhao, and N. Fiedel, “Talm: Tool augmented language models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.12255

    2022 arXiv

  32. [40]

    Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis,

    Y . Lianget al., “Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis,”Intelligent Computing, vol. 3, p. 0063, 2024

    2024

  33. [41]

    Gorilla: Large language model connected with massive APIs,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive APIs,” 2023. [Online]. Available: https://arxiv.org/abs/2305.15334

    2023 arXiv

  34. [42]

    Document expansion by query prediction,

    R. Nogueira, W. Yang, J. Lin, and K. Cho, “Document expansion by query prediction,” 2019. [Online]. Available: https://arxiv.org/abs/1904. 08375

    2019

  35. [43]

    The faiss library,

    M. Douzeet al., “The faiss library,” 2024. [Online]. Available: https://arxiv.org/abs/2401.08281

    2024 arXiv

  36. [44]

    C-pack: Packaged resources to advance general chinese embedding,

    S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-pack: Packaged resources to advance general chinese embedding,” 2023. [Online]. Available: https://arxiv.org/abs/2309.07597

    2023 arXiv

  37. [45]

    Nv-embed: Improved techniques for training llms as generalist embedding models,

    C. Leeet al., “Nv-embed: Improved techniques for training llms as generalist embedding models,” 2024. [Online]. Available: https://arxiv.org/abs/2405.17428 Robin D. Peslis a Ph.D. student at the Institute of Architecture of Application Systems at the Univer- sity of Stuttgart ...

    2024 arXiv

  38. [2021]

    Available: https://arxiv.org/abs/2110.14168

    [Online]. Available: https://arxiv.org/abs/2110.14168

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.