{"id":"b196a487-4dda-4587-9228-fde965163e9f","arxiv_id":"2608.03558","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EffiHolmes combines differential profiling, compressed execution paths, and guided LLM reasoning to localize where to fix repository-level time inefficiencies, beating baseline methods on a new 140-issue benchmark.","lead":"This paper presents EffiHolmes, a system that finds where to fix slow code in large software repositories by comparing profiling traces under normal and scaled workloads, then asking a large language model to reason over the extracted execution paths. It also introduces a new benchmark of 140 real performance issues and reports gains over several existing localization methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison may be confounded: the differential profiling pipeline consumes the issue's reproduction script, so the reported gains could partly reflect test-case access that the static baselines never receive.","rationale":"The reader's weakest assumption concerns benchmark representativeness and the unaudited LLM-negative cases. That is a real external-validity concern, but the 140 instances do plausibly represent developer-reported issues with confirmed fixes, and selection bias does not directly explain the magnitude of the reported gains. The more load-bearing issue is internal to the evaluation: the comparison may be asymmetric in what input each method receives. EffiHolmes's pipeline necessarily consumes a reproduction script to build the differential test pair, and the benchmark pipeline retained exactly those instances that have reproducible test cases. If the baselines only receive the cleaned issue text while EffiHolmes also runs the issue's reproduction script and the differential traces, the claimed 4.29 pp and 15.00 pp improvements conflate the value of runtime evidence with the value of EffiHolmes's own localization method. The paper states the benchmark kept cases reproducible with provided test cases, but never states whether those test cases are given to EffiHolmes, to the baselines, or to both. The ablation study removes execution paths and guided reasoning but always keeps the identified hotspots and hybrid code context, so it cannot isolate the test-pair construction. This concern is concrete, testable, and would not require new data collection to resolve, so the correct handling is to keep the conditional verdict, with the input-controllability condition spelled out precisely. I therefore disagree with the reader's choice of weakest assumption: the selection-bias concern is secondary, while the input-asymmetry concern is primary and directly affects every headline number.","tokens_in":22294,"tokens_out":2178,"duration_ms":17031,"concrete_test":"Re-run the agent baselines (MoatlessTools, SWE-agent, LocAgent) on RepoEffi-Bench after giving each agent the same reproduction script and the same differential trace pair that EffiHolmes uses, with instructions to use them as evidence, and compare their Acc@k against EffiHolmes. If the baselines close most of the gap, the reported gains are largely due to the profiling evidence itself, not EffiHolmes's localization logic; if they do not, the concern is resolved. Also report how many of the 140 instances include a working reproduction script and whether EffiHolmes and the baselines received identical inputs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"EffiHolmes's pipeline takes the issue report plus the repository, but the whole signal-identification stage depends on constructing and executing a differential test pair. The benchmark construction retains only instances where the provided test cases reproduce the slowdown on the base commit and verify the speedup on the fix commit. The paper does not state whether EffiHolmes is given those test cases or a reproduction script, or whether the baseline agents are likewise given reproduction scripts. If EffiHolmes receives the developer-provided test case or the issue-linked reproduction script while retrieval and agent baselines receive only the cleaned issue description, then the comparison measures access to reproduction evidence plus profiling plus guided reasoning, not the profiling-guided reasoning contribution. The motivating example in Section 2 argues that agents fail without runtime feedback, but no experiment controls for this asymmetry. Since the central claim is 'EffiHolmes outperforms baselines', a large part of the gain may come from differential test pair construction and execution, which is not credited to EffiHolmes's localization logic, and the baselines are denied this input. This is a load-bearing confound because it affects every reported number, not just one ablation row. The ablation in Table 2 removes execution paths and guided reasoning but always retains the differential traces, so it cannot separate the value of profiling from the value of the test-pair construction and execution; it also does not show what a baseline would do if handed the same scaled execution trace.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EffiHolmes, an LLM-based framework for localizing the fix locations of repository-level time inefficiency issues. The framework constructs a differential test pair (default vs. scaled workload), profiles both executions with VizTracer, identifies hotspots by differential self-time and normalized differential contribution, extracts and compresses execution paths that connect the entry point to the hotspots, and then uses a guided-reasoning prompt with hybrid code context to rank candidate fix locations. The paper also introduces RepoEffi-Bench, a benchmark of 140 real-world Python inefficiency issues derived from popular data-science repositories, and reports that EffiHolmes outperforms retrieval-, agent-, procedure-, and profiling-based baselines in all 18 metric-by-model evaluation scenarios, with headline gains of 4.29 pp on GPT-5.1 file-level Acc@3 and 15.00 pp on qwen3-4b function-level Acc@5 over the strongest baselines.","tokens_in":22613,"tokens_out":8002,"duration_ms":67408,"significance":"If the results hold, the paper addresses a genuinely underexplored problem: repository-level inefficiency fix localization without stack traces or failing tests. The differential profiling idea is well motivated by the motivating example, and the paper has clear strengths: a reproducible artifact link, an ablation study isolating the contributions of execution paths and guided reasoning, a failure analysis with concrete issue examples, and a benchmark that is more repository-level and leakage-conscious than prior function-level or synthetic benchmarks. The central pipeline is clearly specified. However, several evaluation-protocol issues currently prevent the reported superiority from being interpreted as clean evidence for the framework's localization logic rather than for its access to reproduction artifacts or its benchmark construction choices.","major_comments":[{"comment":"The evaluation may be confounded by unequal access to reproduction artifacts. Section 3.2.1 says the scaled target is inferred 'from the baseline script and the issue report,' and Section 4.2 states that RepoEffi-Bench retains only cases where the slowdown is reproduced and the speedup verified 'using the provided test cases.' Section 4.3, however, says agent and procedure baselines receive only the repository and the cleaned issue description. If EffiHolmes is given the developer-provided test case or reproduction script while baselines are not, the reported gains (e.g., the 15.00 pp function-level Acc@5 gain over Direct Aggregated Profiling on qwen3-4b) include the effect of differential test-pair construction, not just the profiling-guided reasoning. The ablation in Table 2 does not resolve this, because all ablations retain the differential traces. Please state explicitly what reproduction artifacts each method receives, and add a control in which baselines receive the same test case/script or in which EffiHolmes is run without them.","section":"§3.2.1, §4.2, §4.3"},{"comment":"The Direct Aggregated Profiling baseline is not specified. It is the strongest baseline on qwen3-4b and the comparison point for the paper's headline 15.00 pp gain, yet Section 4.3 describes only retrieval-, procedure-, and agent-based baselines; the profiling-based category is introduced only in Table 1 and RQ1. The paper must define how this baseline obtains per-function runtime statistics (which profiler, which prompt template, which model, what post-processing), otherwise the key comparison is not reproducible and its fairness cannot be assessed.","section":"§4.3, Table 1"},{"comment":"The success rate of differential test pair generation is not reported. The pipeline uses a bounded retry budget of five rounds, but the paper never states how often the LLM-inferred scaling target fails to produce a valid pair, what happens when the budget is exhausted, or whether such instances are removed from the evaluation. If failed instances are dropped, the reported Acc@k values are conditional on successful pair generation and overstate the method's coverage. Please report the per-model generation success rate and re-run the evaluation counting generation failures as misses.","section":"§3.2.1"},{"comment":"The RepoEffi-Bench construction may systematically select for issues that are amenable to the EffiHolmes pipeline. Starting from 1,594 issue-PR pairs, the dataset keeps only 140 cases where the slowdown at the base commit and speedup at the fix commit could be reproduced using the provided test cases; the paper also notes that LLM-negative cases were not manually audited. Because EffiHolmes itself requires a baseline script and a differentially scalable workload, the benchmark could be enriched for exactly the conditions the method needs, limiting external validity. Please provide an analysis of the dropped 1,454 cases (e.g., reasons for exclusion, whether they lack test cases or do not scale) and discuss how this affects the generalizability of the reported accuracies.","section":"§4.2"}],"minor_comments":[{"comment":"The EffiHolmes row for qwen3-4b contains a stray '0.00' between the File Acc@1 and File Acc@3 values; the table should be corrected so that File Acc@3 is reported consistently.","section":"Table 1"},{"comment":"Figure 3 shows 'Test Case' as an input to the pipeline, while Section 3.1 states that the framework takes only the issue report and repository as inputs; the role of the test case/reproduction script in the pipeline should be stated unambiguously.","section":"Figure 3"},{"comment":"CoSIL is listed under agent-based methods in Section 4.3 but appears as a procedure-based method in Table 3; the categorization should be made consistent.","section":"§4.3 and Table 3"},{"comment":"The symbol k is used both for the number of selected hotspots and for the line threshold for full code context; these are different parameters and should be renamed to avoid confusion.","section":"§3.2.2 and §3.4.1"},{"comment":"Many reported margins are very small (e.g., 0.71 pp, which is one instance out of 140), but no confidence intervals or significance tests are given; the claim of consistent superiority across all 18 scenarios would be strengthened by statistical analysis or at least a sensitivity discussion.","section":"Table 1"},{"comment":"The '(NoReason)' variants of EffiHolmes and the baselines are not defined in the text; the paper should state exactly which reasoning component is removed and why baselines are also evaluated in this mode.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the artifact link plus the ablation and failure analysis are good signs. My main concern is that the evaluation protocol may give EffiHolmes access to reproduction artifacts that the baselines do not receive; this should be clarified before the comparison can be interpreted. I would accept a revised version that adds the requested controls, reports the differential-pair generation success rate, and specifies the Direct Aggregated Profiling baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, RepoEffi-Bench fills a real gap: 140 real repository-level inefficiency issues with verified reproducers and leak-cleaned descriptions, where existing benchmarks are function-level or synthetic. That alone makes the paper worth reading. Second, the main comparison is not apples-to-apples. EffiHolmes constructs and executes a differential test pair from the issue's reproduction script, while the retrieval and agent baselines are restricted to read-only repository inspection with the cleaned issue text. The stress-test note lands: the reported gains could largely reflect access to an executable reproducer rather than the profiling-guided reasoning the paper credits.\n\nWhat is genuinely good: the pipeline is clearly specified—differential profiling under default vs. scaled input to isolate hotspots, path compression to cut trace noise, and domain-guided LLM reasoning to climb from symptom to fix location. The ablation is internally consistent and shows execution paths carry most of the lift, which is plausible. The authors are honest about limits: they acknowledge off-path fixes, native-code fixes outside Python-level tracing, and the fact that LLM-negative cases in dataset construction were not manually audited. That candor counts.\n\nThe soft spots are real but fixable. The confound is the biggest one, and it affects every headline number. The paper never states whether EffiHolmes is handed the test case from the benchmark instance; the dataset construction section says instances were retained only when provided test cases reproduced the slowdown, and EffiHolmes's differential pair generation clearly uses that artifact. Baselines get only the cleaned issue text. A direct profiling baseline also appears to have execution access, so the internal comparison against aggregated profiling is fairer, but the agent and retrieval comparisons are not. Also missing: a commit hash for the shipped code and dataset, any variance or repeated-run analysis, and a breakdown of how often the five-round test-pair generation fails. These are documentation gaps, not fatal flaws.\n\nWho should read it: anyone working on performance debugging, LLM agents for code, or benchmark construction for software engineering tasks. The benchmark is a contribution that survives even if the method's relative advantage shrinks under a fairer comparison. The paper deserves a serious referee, not a desk reject. I would ask for a revision that (a) states explicitly what inputs each system receives, (b) either gives baselines the same execution access or reframes the claim as \"given an executable reproducer,\" and (c) ships a reproducible package with exact prompts and a commit hash.","headline":"EffiHolmes is a well-built profiling-based localization pipeline with a genuinely useful new benchmark, but the headline comparison with static baselines is confounded because the method gets to run the reproducer and the baselines do not.","tokens_in":23065,"tokens_out":3066,"would_cite":true,"duration_ms":29302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EffiHolmes claims that running a program at two workload scales, tracing the hot path, and then asking an LLM to reason upstream finds where to fix slow code better than retrieval, agent, and profiling baselines on every benchmark scenario.","keywords":["repository-level performance optimization","time inefficiency","differential profiling","fix localization","LLM","fault localization"],"falsifier":"Take a held-out sample of time-inefficiency fixes that live in native code behind Python wrappers or in thin wrapper functions that profilers attribute entirely to built-ins, and run EffiHolmes on them; if its accuracy collapses to retrieval-baseline levels, the claim that differential execution-path evidence suffices to bridge hotspots and fix locations is refuted.","tokens_in":22042,"feed_emoji":"⏱️","tokens_out":8523,"duration_ms":69449,"temperature":0.7,"pith_summary":"This paper is trying to establish that repository-level time-inefficiency issues can be localized to their fix locations even when they produce no test failures or stack-traces, by running the same reproduction under a default and a scaled workload, extracting the execution paths linking the resulting hotspots to the entry point, and guiding an LLM with domain heuristics to reason upstream from symptoms to causes. Single-run profiling drowns inefficiency hotspots in execution noise, the paper argues, while differential profiling reveals which functions grow disproportionately with input scale. Because RepoEffi-Bench contains 140 real developer-reported and developer-fixed issues, the framework's best-or-tied-best result in all 18 evaluation scenarios is presented as evidence that the approach transfers beyond toy settings. A sympathetic reader would care because slow-but-correct code is a major source of cost and operational failure, and no previously existing fault-localization paradigm applies to it.","feed_headline":"Differential profiling finds slow-code fixes, beating LLM agents","feed_subtitle":"Runtime comparison plus path tracing locates where to patch, with top-5 file accuracy up to 90.7 percent.","key_machinery":"The load-bearing mechanism is the differential trace pair and its derived normalized differential contribution (NDC). For each function node $n$, EffiHolmes measures $\\Delta_{\\mathrm{self}}(n)$ as scaled-execution self-time minus default-execution self-time, and $\\mathrm{NDC}(n)=\\Delta_{\\mathrm{self}}(n)/\\sum_m \\Delta_{\\mathrm{self}}(m)$; the top-$k$ nodes by NDC are the inefficiency hotspots. Those hotspots anchor a lower bound, while the reproduction script's entry function anchors an upper bound, and the framework enumerates entry-to-hotspot paths, compresses them by structural call-chain identity, and keeps the representative path with the highest cumulative differential self-time. That compact set of critical paths, together with a hybrid code context (full source for short functions, signature-plus-call-lines for long ones) and a guided-reasoning prompt that favors control-layer functions and requires source-code validation, is what carries the LLM from profiler symptom to fix location.","core_discovery":"The central claim is that a differential-pair profiling pipeline can close the semantic gap between where time accumulates and where a fix belongs. EffiHolmes profiles the same reproduction under two workload scales, ranks functions by normalized differential self-time ($\\mathrm{NDC}(n)=\\Delta_{\\mathrm{self}}(n)/\\sum_m \\Delta_{\\mathrm{self}}(m)$), reconstructs compressed entry-to-hotspot call-chain paths, and prompts an LLM to prioritize upstream control-layer logic while validating candidates against source-code evidence and six measured signals. On RepoEffi-Bench this yields the best or tied-best result in all 18 scenarios (6 metrics × 3 models), including 4.29 percentage-point gains on GPT-5.1 file-level Acc@3 and 15.00 percentage-point gains on qwen3-4b function-level Acc@5 over the strongest baseline.","pith_inferences":["Beyond the paper: the differential workload-scaling idea is language-agnostic and likely extends to memory or I/O inefficiencies, since any resource whose consumption grows superlinearly with input scale can be substituted for wall-clock self-time.","Beyond the paper: the benchmark's filtering stages—LLM classification with no audit of negatives, plus requiring reproducible slowdowns via provided test cases—may make RepoEffi-Bench easier than uncurated issue streams; applying EffiHolmes to non-reproducible or test-less issues would be a stricter check.","Beyond the paper: because the pipeline only ranks candidates on extracted paths, its recall ceiling is set by profiler visibility; adding static taint or native-code instrumentation could recover the off-path fixes the paper identifies as failure modes."],"forward_implications":["A 4B-parameter model using EffiHolmes reaches 83.57% file-level Acc@5 and 70.71% function-level Acc@5, matching or beating baselines that run on models with roughly 8× more parameters.","Execution-path extraction is the primary source of the gain: ablating it drops function-level Acc@3 by 39.29 percentage points on GPT-5.1.","Differential profiling isolates complexity growth that single-run profiles cannot show, so the approach works even when the inefficient operation is tiny relative to background noise.","Guided reasoning and execution paths are complementary: removing both drops function-level Acc@1 to 22.86%, below the least effective single-component ablation.","Function-level localization, the harder developer-facing task, is where EffiHolmes' advantage over retrieval-based methods is largest."],"supporting_citations":[{"why":"Supplies the low-overhead trace profiler that records full call-tree structure and deterministic timestamps for both executions.","marker":"[22]"},{"why":"Establishes that profiler hotspots show where time accumulates, not where issues originate, motivating the guided upstream reasoning.","marker":"[71]"},{"why":"Provides the empirical characterization of real-world performance bugs that motivates differential signal amplification and path extraction.","marker":"[34]"},{"why":"Documents how developers navigate deep call chains to locate performance bugs, informing the design of entry-to-hotspot path reconstruction.","marker":"[12]"},{"why":"Provides the issue-PR collection practice and repository quality criteria adopted to construct RepoEffi-Bench.","marker":"[33]"},{"why":"Justifies the Acc@k metrics by showing practitioners regard the top-5 results as the practical localization target.","marker":"[37]"},{"why":"An agent-based baseline that navigates the repository graph; one of the strongest baselines EffiHolmes must beat.","marker":"[16]"},{"why":"A procedure-based hierarchical localization baseline representative of recent LLM localization pipelines.","marker":"[60]"}],"fun_headline_variants":["Differential profiler outperforms LLM agents on slow-code fixes","Two-scale profiling finds slow-code fixes faster than agents","Differential profiling locates slow-code fixes, beating agent baselines","Profiler-guided LLM pinpoints slow-code fixes with two runs","EffiHolmes: differential profiling beats LLM agents on slow-code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured accuracy transfers to the real world of repository-level time inefficiency issues, because the 140 instances in RepoEffi-Bench are a representative and unbiased sample of that world rather than a sample shaped by the filtering pipeline.","fun_headline_variants_meta":{"raw":{"variants":["Differential profiler outperforms LLM agents on slow-code fixes","Two-scale profiling finds slow-code fixes faster than agents","Differential profiling locates slow-code fixes, beating agent baselines","Profiler-guided LLM pinpoints slow-code fixes with two runs","EffiHolmes: differential profiling beats LLM agents on slow-code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2862,"prompt_tokens":981,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":597,"tokens_out":1881,"duration_ms":12930,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:47:35.902839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out sample of time-inefficiency fixes that live in native code behind Python wrappers or in thin wrapper functions that profilers attribute entirely to built-ins, and run EffiHolmes on them; if its accuracy collapses to retrieval-baseline levels, the claim that differential execution-path evidence suffices to bridge hotspots and fix locations is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the low-overhead trace profiler that records full call-tree structure and deterministic timestamps for both executions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents how developers navigate deep call chains to locate performance bugs, informing the design of entry-to-hotspot path reconstruction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An agent-based baseline that navigates the repository graph; one of the strongest baselines EffiHolmes must beat."}],"review_version":2}