{"id":"b0cfa610-bc24-4495-8760-2b15b4db4fd7","arxiv_id":"2506.15167","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM agent using MCP tunes WS-PSO-CM hyperparameters and reports a minimum sum-rate of 22.28 bps/Hz, 54.33% and 72.61% above human heuristic and single random baselines.","lead":"An LLM agent iteratively tunes the hyperparameters of a UAV trajectory and communication optimization algorithm and reports higher minimum data rates than two baselines. It is a practical demonstration of using large language models to automate wireless system tuning, though the evidence is a single scenario with no error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run, no-seed comparison cannot support the claimed 54.33% and 72.61% gains; stochasticity and unequal Pnum budgets are uncontrolled.","rationale":"The paper builds a plausible engineering pipeline: an LLM agent with MCP iteratively proposes WS-PSO-CM hyper-parameters, and the reported minimal sum-rate rises over six iterations to 22.28 bps/Hz. The central claim, however, is comparative: that these hyper-parameters significantly outperform human heuristics and random generation. The weakest point is the statistical basis of that comparison. WS-PSO-CM is stochastic, yet every reported value in Figs. 4 and 5 is a single number with no seeds or error bars; the human baseline uses 100 particles while the LLM proposals use at most 50; and the random baseline is one draw from a uniform distribution rather than a distribution of draws. Any one of these issues would weaken the claim, and together they leave the headline gains unsubstantiated. This is not an internal inconsistency or a disagreement with consensus; it is a question of whether the evidence supports the stated effect. The proposed repeated-seed test with equal Pnum is a direct, feasible check: if the LLM hyper-parameter set separates from baselines across many seeds, the core claim is supported; if not, the observed gain is likely noise or a compute-budget artifact. The reader's verdict of REJECT is therefore appropriate as the manuscript stands, with a clear path to CONDITIONAL if distributional evidence, matched budgets, and artifacts such as code, prompts, and data are provided. No significant objection beyond this statistical-confound concern was identified, and no change to the reader's verdict is recommended.","tokens_in":9205,"tokens_out":3240,"duration_ms":37552,"concrete_test":"Re-run WS-PSO-CM over at least 30 random seeds for each of the six LLM hyper-parameter sets in Table I and for the two baselines, using identical seed lists and the same system model; report the mean, standard deviation, and 95% CI of the minimal sum-rate for each set. To remove the compute-budget confound, set Pnum equal across all compared sets (e.g., 50 for all), or normalize by wall-clock runtime. If the 22.28 bps/Hz value falls inside the baseline confidence intervals, or if the mean LLM-set performance is not separated from the human-heuristic mean, the claimed gains do not survive. Optionally repeat the full LLM agent loop 3–5 times to verify that the six-iteration trajectory is not one lucky run.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV (Figs. 4–5) reports one minimal-sum-rate value per hyper-parameter setting: proposed 22.28 bps/Hz, human heuristic 14.44 bps/Hz, uniform random 12.91 bps/Hz. WS-PSO-CM is stochastic: the velocity update in Eq. (4a) uses rand1 and rand2 drawn from U[0,1], and the PSO initialization and crossover/mutation operators are randomized. No seeds, repeats, or error bars are reported, so each plotted point could be a favorable or unfavorable draw. The comparison is also unequal: the human baseline uses Pnum = 100 while every LLM proposal in Table I uses Pnum <= 50, so the apparent gain may partly reflect a compute-budget difference rather than hyper-parameter quality. The uniform-random baseline is a single sample, not a distribution, so the 72.61% gain is not a statement about random generation as a method. The paper itself states in Section IV that Grid Search and Bayesian Optimization are excluded, so the broader claim of automatic hyper-parameter tuning is not tested by these two baselines. The load-bearing condition for the central claim is that the reported differences are attributable to the LLM-proposed hyper-parameters and are reproducible; the current evidence does not establish that condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an LLM agent for tuning the hyper-parameters of the WS-PSO-CM algorithm used in radio map-enabled UAV trajectory and communication optimization. The agent is constructed with a Profile that specifies hyper-parameter boundaries, task objective, terminal condition, strategy, and LLM configuration, and it uses the Model Context Protocol (MCP) to interact with two servers: one passing hyper-parameters to numerical software and one managing history. In each iteration, the LLM proposes a hyper-parameter set, WS-PSO-CM is executed, and the resulting minimal sum-rate is fed back to the LLM. Experiments on a 240x400x60 m^3 HITSZ campus scenario with 4 UAVs and 8 UGVs report a minimal sum-rate of 22.28 bps/Hz after six LLM iterations, compared with 14.44 bps/Hz for the human heuristic configuration from the authors' prior work [3] and 12.91 bps/Hz for one uniformly randomly generated configuration, corresponding to claimed gains of 54.33% and 72.61%.","tokens_in":9465,"tokens_out":5455,"duration_ms":53546,"significance":"If the performance gain were established with proper statistical control, the paper would be a useful demonstration of LLM agents for hyper-parameter optimization in wireless communication, with a concrete MCP-based tool integration and an iterative feedback loop. The paper explicitly provides the algorithm description, the profile concept, and the iteration-level hyper-parameter table, which are helpful for replication. However, the central empirical claim is currently supported only by single-run scalar comparisons on one scenario, with no repeated trials, seeds, confidence intervals, or standard HPO baselines; as it stands, the evidence is not sufficient to conclude that the LLM agent significantly outperforms either baseline. The contribution is therefore promising but not yet established.","major_comments":[{"comment":"Every reported minimal sum-rate is a single number, with no seeds, repeated trials, confidence intervals, or significance tests. WS-PSO-CM is stochastic through the random variables rand1 and rand2 in Eq. (4a) and through randomized initialization, crossover, and mutation; the LLM itself (DeepSeek R1, with unspecified temperature) is also stochastic. Consequently the claimed 54.33% and 72.61% gains may be within the run-to-run variation of the stochastic optimizer. The paper should repeat each configuration over many seeds, report mean and variance or boxplots, and perform a statistical test before claiming significance.","section":"Section IV, Figs. 4 and 5"},{"comment":"The comparison is not controlled. The human heuristic baseline uses Pnum=100, whereas every LLM proposal in Table I uses Pnum<=50, so the comparison mixes hyper-parameter quality with a different particle budget. The 'uniform distribution' baseline is a single sampled configuration (Pnum=58, w=0.8765, c1=5.4321, etc.), not a distribution; without defining the sampling ranges and drawing multiple random configurations, the 72.61% gain is not a statement about random generation as a method. The authors should report the hyper-parameter bounds used in the profile, draw a full random baseline with the same number of evaluations, and match computational budgets.","section":"Section IV, baselines"},{"comment":"The authors explicitly exclude Grid Search and Bayesian Optimization because they consider only a few iterations. Since the paper claims automatic hyper-parameter tuning for WS-PSO-CM and frames current methods as heuristic-based, the absence of any standard HPO baseline makes the broader claim untested. At minimum, a random search with the same number of WS-PSO-CM evaluations and a simple Bayesian optimization baseline should be included to position the LLM agent relative to established methods.","section":"Section IV, paragraph beginning 'Since we only consider...'"},{"comment":"The profile is said to specify the boundary of hyper-parameters and LLM configurations, but the paper never reports these boundaries, the DeepSeek R1 model version, temperature, prompt, or the terminal condition. Without these details the experiment is not reproducible and the reader cannot know whether the uniform baseline draws from the same feasible region. Please provide the full profile and LLM settings.","section":"Sections III-B2 and IV"}],"minor_comments":[{"comment":"The phrases 'an Large Language Model' and 'an uniform distribution' should be corrected to 'a Large Language Model' and 'a uniform distribution'.","section":"Abstract and Section IV"},{"comment":"Equation (4a) updates velocity as Vk = omega*Vk + c1*rand1*(pBestk - Vk) + ..., but the second term should almost certainly be (pBestk - Qk), the difference between the personal best and the current position, rather than the current velocity. Please correct the formula or define the notation.","section":"Equation (4a)"},{"comment":"Algorithm 1 line 5 writes 'rhoBestk' while the text uses 'pBestk'; please unify the notation.","section":"Algorithm 1"},{"comment":"There is a duplicated phrase 'scheduling scheduling' in Section II-A, and the list of UGV speeds in Section IV uses inconsistent comma separators.","section":"Section II-A and Section IV"},{"comment":"The y-axis label 'minimal sum-rate' is ambiguous because (3a) maximizes the minimum over UGVs; please clarify whether the plotted quantity is the optimized max-min value.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does one specific thing – uses an LLM agent to propose hyperparameters for WS-PSO-CM – and reports gains of 54% and 72% over human and random baselines. But the numbers rest on a single run with no seeds or error bars, an unequal compute budget, and a random baseline that is one sample. The claim isn't supported as presented.\n\nWhat's actually new: applying the LLM-agent pattern to this particular PSO variant for UAV trajectory/communication. The framework itself (profile, tools, planning, memory, MCP) is standard from the agent literature, and the references to LLM-based wireless design and HPO already exist. So novelty is modest. What the paper does well: the pipeline is clearly described, the iterative loop with MCP is a reasonable way to wrap an optimization algorithm, and the authors are transparent about what they did – they even state in Section IV that Grid Search and Bayesian optimization were excluded. That honesty is good.\n\nSoft spots: all the load-bearing comparisons are statistically weak. WS-PSO-CM is stochastic (Eq. 4a has rand1/rand2; initialization and crossover are randomized), yet every reported number in Figures 4 and 5 is a single value. The human heuristic baseline uses Pnum=100 while the LLM proposals never exceed 50, so the gain could partly reflect compute budget, not hyperparameter quality. The uniform-random baseline is one draw, not a distribution, so the 72.61% figure is not a statement about random generation. No code, data, or prompt templates are provided, making reproducibility impossible. These problems are fixable: multiple seeds, error bars, proper baselines, and artifacts would make the claim credible.\n\nThe citation pattern looks fine; citing [3] as the source of the baseline is appropriate. The math is not new, but that's not a problem.\n\nBottom line: this is a plausible engineering demo, not an established result. As is, I would not accept it, and I don't think it needs a full referee round until the missing statistical support is added. It could serve as a cautionary reading-group example of LLM-agent claims outpacing evidence.","headline":"A workmanlike LLM-agent-for-HPO pipeline whose headline gains rest on a single unseeded run; the idea is modest and the evidence doesn't hold up yet.","tokens_in":10013,"tokens_out":2744,"would_cite":false,"duration_ms":27763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A profile-guided LLM agent iteratively tunes WS-PSO-CM hyper-parameters and, within six iterations, raises the minimal sum-rate from hand-tuned and random baselines to 22.28 bps/Hz.","keywords":["large language model","model context protocol","radio map","unmanned aerial vehicle","hyper-parameter optimization","WS-PSO-CM","particle swarm optimization","UAV trajectory and communication"],"falsifier":"Run the same WS-PSO-CM solver on the same campus scenario with the LLM-recommended settings and each baseline setting across many random seeds; if the averaged minimal sum-rates overlap within their standard deviations, or if the baseline matches 22.28 bps/Hz once given the same particle count and compute budget, the claimed 54.33% and 72.61% gains are not established.","tokens_in":8956,"feed_emoji":"🚁","tokens_out":5927,"duration_ms":55044,"temperature":0.7,"pith_summary":"This paper proposes an LLM agent that automatically tunes the eight hyper-parameters of the WS-PSO-CM algorithm, a warm-started particle swarm optimizer with crossover and mutation used for radio-map-enabled UAV trajectory and communication. It claims that, within six iterations, the agent finds settings that achieve a minimal sum-rate of 22.28 bps/Hz, which is 54.33% higher than the hand-tuned heuristic configuration and 72.61% higher than a uniform-random configuration. The reason this matters is that hyper-parameter selection for such optimizers is currently manual and expertise-heavy; an automated feedback loop that reads the algorithm's fitness output could replace that labor and transfer to other optimization algorithms.","feed_headline":"LLM agent beats hand-tuned UAV optimizer settings by 54%","feed_subtitle":"An iterative LLM agent lifted the minimal sum-rate to 22.28 bps/Hz, a 54.33% gain over hand-tuned settings.","key_machinery":"The load-bearing mechanism is the closed exploration loop around the WS-PSO-CM solver. WS-PSO-CM is a particle swarm optimizer that initializes particles with a convex-optimization warm start, then applies crossover and mutation to maintain diversity, with a fitness function that weights the minimal sum-rate against speed, turning-angle, and building-collision penalties. The LLM agent proposes the eight tunable hyper-parameters (particle count, inertia weight, cognitive and social coefficients, and the four fitness weights), receives the solver's fitness outcome through a Model Context Protocol tool server, and revises its proposal until the terminal condition triggers.","core_discovery":"The paper's central claim is that a profile-guided LLM agent can navigate the hyper-parameter space of WS-PSO-CM by an iterative propose-and-observe loop, without any hand-tuning. The agent's profile fixes the hyper-parameter boundaries, the objective (maximizing the minimal sum-rate), the stopping condition, and a conservative or aggressive optimization strategy; a Model Context Protocol server hands each proposed set to the solver, and the solver's returned minimal sum-rate is fed back as the signal for the next proposal. Across six iterations the agent's proposals improve the minimal sum-rate from 16.33 to 22.28 bps/Hz, compared with 14.44 bps/Hz for the human-heuristic baseline and 12.91 bps/Hz for the uniform-random baseline. The paper interprets this as evidence that LLM agents with PSO and WS-PSO-CM knowledge can act as general high-performance hyper-parameter seekers.","pith_inferences":["The reported 54.33% and 72.61% gains compare against one uniform-random configuration and one heuristic configuration on a single scenario; a natural test extension is to average over many seeds and over several radio-map scenarios before treating the percentages as general.","Because the human baseline uses 100 particles while the LLM proposals cap at 50, an equal-compute comparison would clarify how much of the gain comes from tuning quality rather than from a smaller, cheaper search.","The same feedback loop could be extended to tune not only the hyper-parameters but also the penalty weights and constraint parameters of problem (P0), effectively letting the agent reshape the optimizer's objective during search."],"forward_implications":["Six iterations of LLM-guided proposals suffice to raise the reported minimal sum-rate from 16.33 bps/Hz to 22.28 bps/Hz in the tested scenario.","The LLM-recommended particle counts (40 to 50) outperform a human heuristic that uses 100 particles, so the automated settings also reduce per-run compute.","Hyper-parameter tuning for WS-PSO-CM can in principle be fully automated, needing only an algorithm description, a fitness signal, and a terminal condition.","The same profile-and-tool structure applies to other tuning-hungry optimizers, including multi-UAV coordination, resource allocation, radio map construction, and reinforcement learning tasks, as the paper's conclusion states."],"supporting_citations":[{"why":"Supplies the WS-PSO-CM algorithm, the radio-map UAV trajectory and communication problem, and the human-heuristic hyper-parameter baseline that the proposed agent is compared against.","marker":"[3]"},{"why":"Motivates LLM-based automated exploration of optimization tasks, the capability the proposed agent relies on.","marker":"[12]"},{"why":"Provides the example of LLM-enabled wireless network planning that the profile-and-feedback agent design extends.","marker":"[16]"},{"why":"Shows LLM agents applied to radio map generation and wireless network planning, placing the proposed tuning agent in that line of work.","marker":"[19]"}],"fun_headline_variants":["LLM agent beats human-tuned UAV optimizer by 54%","Profile-guided LLM agent lifts UAV sum-rate 54% over heuristics","LLM agent tunes UAV hyperparameters: 54% better than humans","LLM agent outperforms hand-tuned UAV settings in hyperparameter search","LLM agent finds better UAV optimizer hyperparameters without human tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion depends on the assumption that the reported single-run results on one campus scenario are representative enough to attribute the higher minimal sum-rate to the LLM agent's tuning rather than to run-to-run randomness or unequal compute budgets.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent beats human-tuned UAV optimizer by 54%","Profile-guided LLM agent lifts UAV sum-rate 54% over heuristics","LLM agent tunes UAV hyperparameters: 54% better than humans","LLM agent outperforms hand-tuned UAV settings in hyperparameter search","LLM agent finds better UAV optimizer hyperparameters without human tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3402,"prompt_tokens":976,"completion_tokens":2426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2332}},"tokens_in":592,"tokens_out":2426,"duration_ms":17719,"temperature":1.0,"reasoning_tokens":2332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:41:43.195875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same WS-PSO-CM solver on the same campus scenario with the LLM-recommended settings and each baseline setting across many random seeds; if the averaged minimal sum-rates overlap within their standard deviations, or if the baseline matches 22.28 bps/Hz once given the same particle count and compute budget, the claimed 54.33% and 72.61% gains are not established.","supporting_citations":[{"cited_title":"Mathematical discoveries from program search with large language models,","cited_arxiv_id":null,"evidence_quote":"Motivates LLM-based automated exploration of optimization tasks, the capability the proposed agent relies on."},{"cited_title":"Large language model agents for radio map generation and wireless network planning,","cited_arxiv_id":null,"evidence_quote":"Shows LLM agents applied to radio map generation and wireless network planning, placing the proposed tuning agent in that line of work."}],"review_version":2}