{"id":"22f284b1-0a80-4ad7-8f12-8bf0d77a92a6","arxiv_id":"2511.00651","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent LLM system with a fine-tuned small language model as solution planner claims 6× faster and 10% more accurate telecom troubleshooting, but the evidence is internal and partly circular.","lead":"The paper describes a multi-agent AI system for telecom network troubleshooting, with a large language model coordinating agents and a fine-tuned small language model generating remediation plans. It reports 6× faster and 10% more accurate troubleshooting than human engineers, but the evidence is internal and lacks a verifiable baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 6× time reduction and 10% accuracy gain rest entirely on an undescribed human baseline: no task set, timing protocol, or accuracy rubric is reported, so the headline result is not reproducible.","rationale":"The reader's weakest assumption is exactly the missing human-baseline measurement methodology in Section IV.A.3. I read the paper in good faith: it describes an interesting MAS/SLM integration, and the qualitative vignettes in IV.A.1 and IV.A.2 show plausible workflows. However, the paper's strongest claim is quantitative, and all quantitative comparison depends on a baseline comparison that is never specified. I also noticed that the SLM fine-tuning evaluation uses RAGAS/format rewards both as training reward and as the reported metric (Section IV.B.3), which is a real circularity, but the human-baseline gap is the more load-bearing problem because it directly supports the headline 6×/10% conclusion. The reader recommended REJECT; my independent stress-test reaches the same conclusion, so the verdict should remain unchanged.","tokens_in":7892,"tokens_out":4876,"duration_ms":52598,"concrete_test":"Obtain the evaluation protocol that produced Figures 3 and 4: the exact number and list of fault cases, human participant characteristics, timing definitions, and accuracy scoring rubric. Then independently run the system and a matched human-engineer group on the same fixed set of faults (same alarms, logs, and performance data) with a pre-registered rule for when troubleshooting stops. Compare mean time per node and accuracy. If the protocol cannot be supplied, or the ratio and accuracy difference do not reproduce, the headline 6×/10% claim is unsupported and should be treated as such.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim (Section IV.A.3; Conclusion) is that the MAS 'achieves a six-fold reduction in mean troubleshooting time per node and a 10% improvement in accuracy' relative to human engineers, as shown in Figures 3 and 4. This is the central quantitative benefit of the paper. The only evidence is the figures themselves; the text does not define how 'mean troubleshooting time per node' was measured, which phases were included (e.g., plan generation, SME approval, execution, root-cause analysis), how the human baseline was collected (number of engineers, their seniority, number of faults, domains), whether the same fault cases were presented to humans and the MAS with identical alarms/logs/PM data, or how 'accuracy' was scored. Without a documented comparison protocol, the 6× and 10% figures are not interpretable as properties of the proposed system. The apparent comparison could be an artifact of case selection, human measurement conditions, or a mismatch between the automated count and manual work. This is an internal-evidence gap, not a philosophical disagreement: the paper's own conclusion is stated as a quantified superiority, and the quantification is unverifiable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent system (MAS) for automated telecom network troubleshooting, combining LLM-based orchestration with a fine-tuned small language model (SLM) solution planner. The SLM is fine-tuned via SFT followed by GRPO-based RFT, using RAGAS-style and format rewards, and is integrated with a HippoRAG knowledge graph built from proprietary troubleshooting documents. The system is evaluated in RAN and Core network domains, with claimed outcomes of a six-fold reduction in mean troubleshooting time per node and a 10% accuracy improvement over human engineers (Section IV.A.3, Figures 3–4). Additional experiments report reward improvements for the fine-tuned 8B model (Table I, Figures 5–6). The paper's conclusions rest on these two sets of quantitative claims.","tokens_in":8205,"tokens_out":5958,"duration_ms":66380,"significance":"If properly validated, the proposed architecture is a relevant contribution: it addresses cost, privacy, and scalability concerns in telecom operations by combining LLM orchestration with a fine-tuned SLM, and it incorporates practical details such as LoRA, GRPO, and multi-GPU RFT. The system design is plausible and the integration of a knowledge-graph RAG with an agentic workflow is of interest to the community. However, the current evidence is insufficient to support the paper's central claims. The human-baseline comparison is undocumented, and the SLM evaluation is circular because it uses the same reward functions employed during training as the evaluation metrics. Strengths of the manuscript include the detailed description of the MAS components, the reward design, and the practical engineering choices; these do not compensate for the missing rigorous evaluation.","major_comments":[{"comment":"The headline claim of 'six-fold reduction in mean troubleshooting time per node and 10% improvement in accuracy' is not supported by the described methodology. No human-baseline collection protocol is reported: the number and seniority of engineers, the fault set and domains, the phases included in 'troubleshooting time' (plan generation, SME approval, execution, RCA), whether the same alarm/log/PM data were presented to humans and the system, and the accuracy rubric are all unspecified. There are no trial counts, confidence intervals, or error bars. The figures are referenced but not described in terms of axes, units, or N. This is load-bearing because the paper's central conclusion rests on these comparative numbers, which cannot be reproduced or verified as written.","section":"Section IV.A.3, Figures 3–4, Conclusion"},{"comment":"The fine-tuning evaluation is circular. The metrics reported in Table I (RAGAS Rewards, Format Rewards, Total Rewards) are the same reward functions used as the training signal in the GRPO-based RFT described in Section IV.B.2. An improvement on the training objective is expected and does not demonstrate that the fine-tuned SLM produces better troubleshooting plans. The claimed reduction in standard deviation is also measured on these same reward signals. To support the conclusion that the SLM is 'domain-grounded' and 'actionable,' the authors need an independent evaluation—e.g., SME scoring of generated plans, exact-match with gold steps, or end-to-end task success on held-out faults.","section":"Section IV.B.3, Table I, Figures 5–6"},{"comment":"The experimental description for the MAS is anecdotal. The text walks through only two example faults (Input Power Failure in RAN and PDU session degradation in Core) without specifying how many cases were run, how alarms/KPIs were selected, or how outputs were scored. Section IV.A.3 states benefits but provides no quantitative experimental design. This makes it impossible to judge robustness or generalizability. Please provide a systematic evaluation protocol, including dataset statistics, the exact metric definitions, and statistical significance.","section":"Section IV.A.1–IV.A.2"}],"minor_comments":[{"comment":"Reference [18] is cited for RAGAS but points to 'DecodingTrust' (Li et al.); the RAGAS paper (Es et al., arXiv:2309.15217) should be cited. Please check all references for accuracy.","section":"References [18]"},{"comment":"References [2] and [10] are duplicate entries for the same 'Generative AI' paper. Please consolidate.","section":"References [2] and [10]"},{"comment":"Reference [14] for HippoRAG appears to point to the follow-up 'From RAG to Memory' rather than the original HippoRAG paper. Clarify which algorithm/version is actually used.","section":"Reference [14]"},{"comment":"The sentence '1000 training steps executes faster and converges more smoothly' is grammatically awkward. Also, 'proved more effective' should be 'proved to be more effective'. Please revise for clarity.","section":"Section IV.B.1"},{"comment":"Define 'per node' and 'accuracy' explicitly. The current text leaves the units and the accuracy metric ambiguous; this is important for interpreting the headline numbers.","section":"Section IV.A.3"},{"comment":"The figures are not included in the submitted text and no axes or captions are described. Please ensure they include error bars, sample sizes, and clear definitions of the metrics plotted.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The central quantitative claims are presently unsupported by the described evaluation. The missing human-baseline protocol and the circular use of training rewards as evaluation metrics are serious issues that would need to be addressed before publication. If the authors can provide a rigorous independent evaluation—ideally with a documented human comparison, held-out test instances, and non-circular metrics—the work may be worth reconsidering. The reference errors also suggest the manuscript needs careful proofreading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it reads like a solid Ericsson engineering write-up that got oversold in the abstract. The actual contribution—an LLM-orchestrated multi-agent system with a HippoRAG-backed RCA agent and a GRPO-RFT fine-tuned SLM solution planner—is real, and the paper does a decent job describing the architecture, the reward design, and the engineering pain points (chunking, LoRA rank, memory, training time). That part is genuinely useful for anyone trying to build a similar system in a resource-constrained industrial setting.\n\nThe soft spots are exactly where the stress-test note points. The headline claim—6× faster troubleshooting and 10% more accurate than human engineers—has no measurement methodology attached. No task set, no timing protocol, no accuracy rubric, no details on how the human baseline was collected, no error bars, no number of trials. Figures 3 and 4 show bars, but the text never tells you what they are. As written, those numbers are not interpretable, and the reader is right to flag them as unsupported. This is not a philosophical disagreement; the conclusion states a quantified superiority that the paper does not evidence.\n\nThe second issue is the circularity in the SLM evaluation. The fine-tuned model is judged on RAGAS and format rewards, and those exact rewards were used as the training signal during GRPO. So the improvement from 5.96 to 10.51 in Table I is partly the model getting better at the metric it was optimized against. That does not mean the fine-tuning is worthless—the reduced standard deviation and the qualitative example in Figure 7 show something real happened—but the evaluation axis is not independent. A human-rated or task-level evaluation on held-out troubleshooting problems would have been far more convincing.\n\nIf the authors removed the 6× and 10% claims or recast them as preliminary observations, this would be a solid industry-practice paper. As it stands, the central quantitative assertion is not reproducible from the text. That gap is large enough to justify a reject at most venues, though it is a conditional reject rather than a reject on the merits of the system itself.","headline":"Credible industrial system paper whose headline speedup/accuracy numbers are not backed by any documented measurement methodology, and whose SLM evaluation is partly circular; the engineering is worth reading but the claims need major revision.","tokens_in":8669,"tokens_out":1813,"would_cite":false,"duration_ms":23227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent AI system cuts telecom troubleshooting time sixfold and improves accuracy by 10 percent.","keywords":["multi-agent system","small language model","network troubleshooting","reinforcement fine-tuning","retrieval-augmented generation","knowledge graph","radio access network","core network"],"falsifier":"Re-run the same set of fault cases with a panel of engineers and with the MAS, using identical stopping criteria and a pre-registered accuracy rubric; if the MAS does not show roughly a 6× time reduction and 10% accuracy advantage, the paper's headline claims are not supported.","tokens_in":7834,"feed_emoji":"📡","tokens_out":7761,"duration_ms":77662,"temperature":0.7,"pith_summary":"Telecom network troubleshooting still leans on human experts manually correlating alarms, logs, and counters—slow, costly, and hard to scale. This paper argues that a multi-agent system, with a large language model coordinating specialized agents, can automate the whole loop, and that the hard part—generating trustworthy repair plans—can be handled by a small language model fine-tuned on internal troubleshooting documents. The authors report a sixfold reduction in mean troubleshooting time per node and a 10% accuracy gain in trials on radio access and core networks. They also show that the fine-tuned 8-billion-parameter model produces stepwise plans with specific counters and checks, at quality comparable to a much larger commercial model. If the claims hold, operators could shift from expert-driven triage to AI-assisted automation while keeping sensitive telemetry private and cutting reliance on expensive external AI services.","feed_headline":"AI agents cut telecom troubleshooting time sixfold","feed_subtitle":"A small tuned model plans repairs as well as a large API-based model, keeping telemetry private and costs low.","key_machinery":"The load-bearing mechanism is the agentic workflow, specifically the solution-planner agent: a small language model fine-tuned with supervised plus reinforcement fine-tuning, guided by reward functions that check whether answers are complete, relevant, and grounded in retrieved evidence. The planner is coupled to a knowledge-graph retrieval layer built from internal performance-management counters, fault alarms, and configuration documents, so every generated step references real network objects. An LLM orchestrator delegates to the planner, a data retriever, a root-cause analyzer, an executor, and a display agent in a reason-act loop, with a human-in-the-loop approval before execution. The","core_discovery":"The authors' central claim is that end-to-end automated troubleshooting can be assembled from an LLM orchestrator plus a small, domain-tuned 'solution planner' that converts retrieved troubleshooting knowledge into executable stepwise plans. The planner is fine-tuned in two stages—supervised training followed by reinforcement fine-tuning with rewards for completeness, relevance, and groundedness to the retrieved context—and is backed by a knowledge graph of performance, alarm, and configuration documents. In evaluations covering a RAN power-supply fault and a core-network PDU-session degradation, the system produced plans specifying the exact counters and alarms to check, localized faults (e","pith_inferences":["The biggest unstated assumption is the human baseline: the reported 6×/10% figures would carry much more weight if the comparison were run on identical fault cases with a pre-registered accuracy rubric, rather than the summary figures presented.","The fine-tuning approach suggests a general template: for any regulated industry with proprietary operational manuals, a small model fine-tuned on those manuals and grounded by retrieval could reach parity with frontier LLMs at a fraction of the cost—testable by applying the same pipeline to, say, datacenter or grid operations.","A cost-per-incident analysis, comparing the MAS's compute and human-oversight costs against a fully human process, would make the business case concrete; the paper reports time and accuracy but not total cost or error-consequence severity.","One could test how the system degrades when retrieval documents are missing or outdated, since the groundedness reward assumes the knowledge graph reflects current network configurations."],"forward_implications":["Operators could turn first-line troubleshooting into an automated triage service, reserving human experts for edge cases, reducing operational cost and time-to-restore.","A fine-tuned small model can replace per-query calls to large external LLM APIs for plan generation, enabling on-premise deployment that keeps sensitive telemetry inside operator environments.","The reward design—completeness, relevancy, groundedness—provides a reusable recipe for grounding language models in other documentation-heavy operational domains.","Because the system produces stepwise plans with named counters and alarms, its outputs are inspectable by experts, which supports adoption in safety-conscious network operations.","The same architecture could extend beyond the two fault families tested (power failures and PDU-session degradation) to other alarm classes, provided retrieval documents exist."],"fun_headline_variants":["Fine-tuned small models slash telecom troubleshooting time sixfold","Multi-agent system automates telco fault fixes sixfold faster","Small tuned models plan telecom repairs as well as big LLMs","LLM orchestrates agents and small tuned planner for telco fixes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 6× speedup and 10% accuracy gain are computed against a human-engineer baseline that the paper never describes—how the human times were measured, how accuracy was scored, or whether the same fault cases were used for both the system and the humans.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned small models slash telecom troubleshooting time sixfold","Multi-agent system automates telco fault fixes sixfold faster","Small tuned models plan telecom repairs as well as big LLMs","LLM orchestrates agents and small tuned planner for telco fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":1952,"prompt_tokens":751,"completion_tokens":1201,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1130}},"tokens_in":495,"tokens_out":1201,"duration_ms":13946,"temperature":1.0,"reasoning_tokens":1130,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:27:53.271836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same set of fault cases with a panel of engineers and with the MAS, using identical stopping criteria and a pre-registered accuracy rubric; if the MAS does not show roughly a 6× time reduction and 10% accuracy advantage, the paper's headline claims are not supported.","supporting_citations":[],"review_version":1}