{"id":"ef4da203-c683-4232-b93a-824439fc2c2e","arxiv_id":"2509.20172","paper_version":7,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Even the strongest open-source LLM generates fully correct API invocation code on only 40% of WAPIIBench tasks, versus 77% for GPT-4o.","lead":"WAPIIBench is a new benchmark and test pipeline for how well large language models can write JavaScript code that calls web APIs. Across 395 tasks from four real APIs, the best open-source model is correct at most 40% of the time, while GPT-4o reaches 77% when the endpoint is provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-match ground truth may understate true correctness; if any alternate valid configuration exists, the ≤40% claim for Code Llama 70B (argument completion) fails. A human re-judgment of the 237 non-matching outputs can settle it.","rationale":"The reader's CONDITIONAL verdict focuses on the unique-configuration assumption; I agree this is the load-bearing point. The paper is transparent and the dataset is valuable, but the headline claim is a quantitative statement about how many tasks open-source models can solve, and the measurement instrument defines 'solved' as exact match to a single curated configuration. If the true set of acceptable configurations is larger, the reported 30%/40% are lower bounds, and the 40% ceiling could be exceeded by the best model. My proposed human re-judgment directly tests whether the benchmark's strictness changes the ordinal conclusion. I did not elevate the limited-API or no-confidence-interval concerns: they affect generalizability and precision of point estimates but do not threaten the internal validity of the '≤40%' statement for WAPIIBench. The human-rater test is cheap, decisive, and consistent with the authors' own acknowledgment that some constraints live in free-text and are not captured automatically.","tokens_in":26742,"tokens_out":5626,"duration_ms":41794,"concrete_test":"Select all Code Llama 70B argument-completion outputs that did not exactly match the ground-truth config (approximately 237 samples). For each, present two independent software engineers the task comment t and the relevant OpenAPI spec (but not the ground-truth c), and ask: 'Does this generated request solve the task? Treat harmless extra optional parameters and equivalent value representations as correct if they do not conflict with the spec or the task.' If at least one output is judged correct by both raters, Code Llama's true solve rate exceeds 40%, invalidating the paper's headline claim; report the inter-rater agreement and the corrected rate. If zero are judged correct, the exact-match assumption is supported for this critical case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('no open-source model solves more than 40% of tasks') rests on the correctness metric in §II-D, which compares the captured request configuration c' to the hand-curated ground-truth c via exact/element-wise match (Table II, 'Correct implementations'). The authors manually enforced a unique c per task (§II-A), but this only establishes that their own c is one possible solution; it does not establish that no other request configuration solves t. Real API requests admit many equivalent formulations: optional parameters not mentioned in t may be added without changing the outcome, parameter values may be expressed in semantically equivalent ways, and headers/params ordering or omitted default-valued parameters can vary. §III-C.2 acknowledges that free-form textual constraints in the API spec are not captured by the automated analysis. Consequently, the exact-match rate is a lower bound on true functional correctness. For Code Llama 70B in argument completion, the reported rate is 0.40 (≈158/395); a single additional truly-correct sample among the 237 'incorrect' outputs would push it above 40%, directly falsifying the headline claim. The paper therefore needs an independent human-validated criterion of 'solves t' to confirm that the benchmark's exact-match metric is not systematically penalizing valid alternatives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WAPIIBench, a benchmark with 395 samples covering four real-world OpenAPI-based APIs (Asana, Google Calendar, Google Sheets, Slack). Each sample pairs a natural-language task t with a hand-curated ground-truth request configuration c. The authors evaluate 21 open-source LLMs and two commercial models on JavaScript/Axios code completion in two setups (full completion and argument completion). Generated code is executed in a mock environment and the captured request configuration c' is compared to c via exact and element-wise metrics (Table II). The main empirical claim is that no evaluated open-source model solves more than 40% of tasks: Code Llama 70B reaches 30% (full completion) and 40% (argument completion), while GPT-4o reaches 60% and 77%, respectively. The paper also analyzes error types such as hallucinated endpoints, illegal arguments, and missing arguments, and releases the dataset and evaluation pipeline as open-source artifacts.","tokens_in":26964,"tokens_out":3997,"duration_ms":29755,"significance":"If the central claim is accepted, WAPIIBench is a valuable, first-of-its-kind benchmark for an underexplored task. The experimental design is generally careful: code is executed safely in a controlled environment, metrics distinguish total vs. executable samples, the dataset was manually curated (9 automated fixes, 58 manual fixes), and the dataset-generating model (Gemini 1.5 Pro) is excluded from the reported results. The public artifact and detailed per-API tables further support reproducibility. The central weakness is the reliance on exact configuration match as a proxy for functional correctness, which the authors acknowledge but do not validate against human judgment; this matters because the headline claim is stated in terms of solving tasks.","major_comments":[{"comment":"The 'Correct implementations' metric is an exact match between the captured configuration c' and the hand-curated ground-truth c (Table II). The manual curation in §II-A establishes that the curated c is one correct configuration for each t, but it does not establish that it is the unique one. Since the headline claim (no open-source model >40%) is a binary statement over this metric, the lack of a human-validated re-judgment of non-matching outputs makes the reported rates lower bounds rather than demonstrated upper bounds. For Code Llama 70B in argument completion, correct implementations (t)=0.40 (≈158/395); re-judging the 237 non-matching outputs by human raters (with t and the OpenAPI spec) would determine whether any are functionally correct but configuration-divergent. Without this validation, the claim 'only 40% completely right' is not fully supported. A human re-judgment is a c","section":"§II-D, Table II; §II-A"},{"comment":"The paper describes the evaluation as measuring 'functional correctness' (§II-C), but the automated analysis only captures the outgoing request configuration, not response handling, and it cannot enforce usage constraints that appear only in free-form textual descriptions in the API specification (§III-C.2). This gap is acknowledged in the limitations, but the abstract and conclusion use stronger language ('solve', 'completely right'). The metric is more precisely 'configuration-exact match' than 'functional correctness'. Either the terminology should be softened, or the human re-judgment proposed above should be used to show that exact match is necessary and sufficient for the task as stated. This is not just a wording issue: it affects how readers interpret the central quantitative claims.","section":"§II-C and §II-D; §III-C.2"}],"minor_comments":[{"comment":"The sentence 'None of the evaluated open-source models was able to solve more than 40% of the tasks' should be qualified with 'under exact configuration matching' to avoid overstating the result.","section":"Abstract"},{"comment":"Some cells in the provided full result tables appear as concatenated values (e.g., '0.001.00' and '0.001' in the rendering). Ensure the camera-ready tables are properly typeset and each numeric entry is clearly separated.","section":"Appendix F, Tables VI/VII"},{"comment":"The dataset's synthetic nature—optional parameters used sparingly, placeholder values—limits transferability to real-world API integration, as the authors note. Consider stating more explicitly in the conclusion that the headline percentages apply to this synthetic task distribution, not to web API integration in general.","section":"§III-C.3"},{"comment":"The related work claim 'We are not aware of an evaluation method able to provide results as fine-grained as ours' is strong. It is acceptable as a claim, but consider softening to 'to our knowledge' to match the surrounding style.","section":"§IV"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about exact-match ground truth lands: the central claim depends on a binary metric that has not been validated against human judgment of the non-matching outputs. The paper is otherwise methodologically sound and the authors are transparent about limitations. A human re-judgment of the 237 non-matching argument-completion outputs for the best open-source model would resolve the main risk. If the authors add this validation or carefully re-scope their claims, the paper would be much stronger and likely acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you'll see from Maninger et al. is the first execution-based benchmark I know of for general OpenAPI web API invocation code in raw JavaScript/Axios. WAPIIBench gives you 395 task–configuration pairs across four real APIs, a mock execution pipeline that captures the outgoing request, and a fine-grained set of metrics that go beyond full-match correctness. The manual curation is real work: nine automated fixes, 58 manual fixes, and they deliberately excluded Gemini from the reported results because it generated the dataset. That's the right call.\n\nThe central empirical finding—that no open-source model cracks 40% of tasks while GPT-4o lands at 60–77%—is probably in the right direction. The models hallucinate endpoints and arguments, and the gap is large enough that a bit of measurement noise won't erase it.\n\nThat said, the exact-match correctness metric is the soft spot. The authors enforce a unique ground-truth configuration per task, but that only proves their configuration is one solution, not that it's the only one. Real API requests can be written with extra optional parameters, semantically equivalent values, or omitted defaults, and the prompt actually tells the model not to include unnecessary parameters, so any such addition counts as a mismatch. The stress-test note is fair: for Code Llama 70B on argument completion, a single additional truly-correct output among the 237 non-matches would push the rate above 40% and falsify the headline as stated. The paper's own limitation section (III-C.2) acknowledges that free-form textual constraints aren't captured by the automated analysis, so this isn't a contrived concern. I'd want a human re-judgment of a sample of the non-matching outputs before treating the exact 40% figure as precise.\n\nThe other gaps are more minor, and largely disclosed: only four APIs, synthetic tasks biased towards optional-parameter sparsity, greedy decoding without reruns, and no confidence intervals. These limit generalization but don't undermine the main story.\n\nWho should read this: anyone working on LLM-based code generation for web APIs, API misuse detection, or benchmark design for code execution. It's a solid baseline that others will build on, and the artifact commitment gets real credit.\n\nRecommendation: send it to peer review. A serious referee should push for a small human validation study on the non-exact outputs and some variance reporting, but the benchmark and the core empirical finding deserve to be in the literature.","headline":"A genuinely useful execution-based benchmark for LLM web API integration code, with a real capability gap finding that is probably robust in direction but fragile at the exact 40% threshold; worth engaging seriously.","tokens_in":27519,"tokens_out":1720,"would_cite":true,"duration_ms":27885,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Among 21 open-source LLMs, none solves more than 40% of web API invocation tasks.","keywords":["large language models","code generation","web API integration","benchmark","OpenAPI","hallucination","functional correctness","Axios"],"falsifier":"Give each open-source model the 395 WAPIIBench tasks with the relevant OpenAPI specification excerpt appended to the prompt (without the ground truth). If the correct-implementation rate stays at or below 40%, the paper's ceiling reflects a task-level difficulty independent of closed-book recall; if it rises well above 40%, the ceiling is specifically a memorization limitation, and the benchmark's headline result would be expected to dissolve once context is supplied.","tokens_in":26586,"feed_emoji":"🐛","tokens_out":12057,"duration_ms":71024,"temperature":0.7,"pith_summary":"WAPIIBench asks large language models to complete a short JavaScript program that must invoke a web API endpoint described by a natural-language comment, then executes the generated code in a sandbox and compares the captured HTTP request—method, URL, headers, query, and body—against a hand-curated ground truth. The central empirical finding is that this task is far harder for current models than local code completion: the best open-source model produces a fully correct request 30% of the time when it must choose the endpoint, and 40% when the endpoint is supplied, while the strongest commercial model reaches 60% and 77%. Models frequently hallucinate URLs and parameter names, with 14–39% of generated URLs being illegal and argument precision and recall lagging well behind what one would expect from a capable code model. The paper argues that web API integration is a distinct benchmark category because correctness depends on coordinating multiple argument locations against an externally documented specification, not on matching a single function signature.","feed_headline":"Open-source LLMs solve at most 40% of web API tasks","feed_subtitle":"A 395-task benchmark that executes generated code shows hallucinated endpoints and arguments are the norm.","key_machinery":"The central object is the request configuration c: a JSON structure modeling the HTTP method, full URL, headers, query parameters, and request body that a given task requires. The pipeline's novel step is its mock adapter, which executes the generated JavaScript in a sandbox, intercepts the outgoing Axios request, and serializes it into a captured configuration c′. Correctness is then computed by comparing each element of c′ with the ground-truth c (exact per-field match) and by validating c′ against the OpenAPI specification to separate 'illegal but plausible' from 'legal but wrong' outputs. This element-wise comparison yields the fine-grained metrics—correct URL, illegal URL, method correc","core_discovery":"The authors claim that current open-source LLMs, when prompted with only the API name and a task comment, do not have reliable memorized knowledge of how to call web APIs. Across 395 tasks drawn from four real-world OpenAPI-specified services, the best open-source model (Code Llama 70B) produces an HTTP configuration that exactly matches the ground truth on 30% of full-completion tasks and 40% of argument-completion tasks; the strongest commercial model, GPT-4o, scores 60% and 77%. The error analysis shows the failures are not random syntax errors but systematic missteps: models pick URLs that are not defined in the specification (up to 39% of executable outputs), pass arguments the endpoint","pith_inferences":["Editorial extension: The reported percentages measure only the outgoing request; the paper explicitly does not evaluate response handling. An end-to-end task that also required parsing the response would likely lower the success rates, so the 30–40% figures should be read as an upper bound on full integration correctness.","Editorial extension: Because the ground truth is a single exact configuration, two models that both solve a task but choose different valid parameter placements or equivalent header syntax would be scored as wrong. A human-equivalence rejudging of a sample of mismatched outputs would quantify how many 'incorrect' answers are actually acceptable requests.","Editorial extension: The benchmark only uses base (non-instruction-tuned) models and no in-context specification. Re-running the same pipeline with instruction-tuned models or with the OpenAPI spec in context would separate memorization failure from instruction-following failure; if the latter is the bottleneck, the 40% ceiling may not transfer to modern chat-based assistants."],"forward_implications":["If a model cannot pick a correct endpoint from memory, API-integration assistants that rely on closed-book generation will keep producing failed requests; injecting the relevant OpenAPI spec via retrieval or explicit context becomes the obvious remediation, and the benchmark provides a way to measure its effect.","The two experimental setups give a clean decomposition: endpoint selection versus argument assembly. Because argument completion is substantially easier for all models, future progress can be attributed to one of these skills rather than confounded overall scores.","The fine-grained error metrics (illegal URLs, illegal arguments, missing arguments) define a concrete checklist for constrained decoding or static verification: a guard that rejects endpoints not present in the specification, or arguments not in the endpoint's schema, would immediately eliminate a large share of the observed failures.","Because the dataset contains one sample per endpoint and the evaluation is fully automated, the benchmark can be extended to new APIs, languages, or HTTP libraries without re-designing the correctness analysis; the configuration abstraction carries over."],"fun_headline_variants":["Open-source LLMs solve at most 40% of web API tasks","LLMs hallucinate endpoints: best open model hits 40% ceiling","Web API codegen benchmark: open LLMs max out at 40%","39% of LLM web API outputs use undefined URLs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire comparison rests on the premise that each task has exactly one correct HTTP request configuration and that exact match against that configuration is both necessary and sufficient for functional correctness; constraints that live only in prose, or tasks with multiple valid requests, would break the scoring.","fun_headline_variants_meta":{"raw":{"variants":["Open-source LLMs solve at most 40% of web API tasks","LLMs hallucinate endpoints: best open model hits 40% ceiling","Web API codegen benchmark: open LLMs max out at 40%","39% of LLM web API outputs use undefined URLs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1200,"prompt_tokens":661,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":405,"tokens_out":539,"duration_ms":6691,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T15:12:42.299124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give each open-source model the 395 WAPIIBench tasks with the relevant OpenAPI specification excerpt appended to the prompt (without the ground truth). If the correct-implementation rate stays at or below 40%, the paper's ceiling reflects a task-level difficulty independent of closed-book recall; if it rises well above 40%, the ceiling is specifically a memorization limitation, and the benchmark's headline result would be expected to dissolve once context is supplied.","supporting_citations":[],"review_version":1}