{"id":"3b616f5f-6045-47b6-a161-11b598bc2331","arxiv_id":"2501.11031","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An adaptive log analysis framework that routes uncertain SLM predictions to an LLM with error-case prompts, improving accuracy and cutting LLM cost by about 73%.","lead":"AdaptiveLog combines a small fine-tuned language model and a large language model to analyze system logs, sending only uncertain cases to the expensive large model. The framework reports state-of-the-art accuracy on six log analysis tasks while cutting large-model queries by about 73%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uncertainty routing rule in §3.3 is inverted as written: it counts low-variance observations as 'uncertain,' which contradicts the reported easy/hard split and must be resolved before the 73% cost claim can be accepted.","rationale":"The reader's weakest assumption is exactly the sign convention of the Monte-Carlo-dropout count in Section 3.3, and I agree that it is the most load-bearing point. The central claim is that AdaptiveLog routes only genuinely hard samples to the LLM, cutting LLM cost by 73% while improving accuracy. That claim requires alpha to count genuinely unstable (high-variance) predictions. The written definition of alpha counts the opposite, so either the paper text is wrong or the reported easy/hard separation cannot be explained by the stated algorithm. This is an internal inconsistency, not a disagreement with consensus. The released code can settle it: a one-line comparison determines whether the textual inversion is real or merely a typo. If the code uses the inverted condition, the reported results are not reproducible from the paper and the framework's main benefit is unsubstantiated. If the code uses the correct condition, the concern reduces to a documentation error, and the conditional acceptance can proceed. I would keep the reader's CONDITIONAL verdict: the framework is plausible and extensively evaluated, but the routing rule needs verification before the cost-accuracy trade-off can be accepted. I do not weight the secondary issues (missing 'Table ??' reference, typos, missing baselines) as heavily, because they are correctable and do not directly undermine the mechanism.","tokens_in":30672,"tokens_out":6242,"duration_ms":73126,"concrete_test":"In the released repository (github.com/LeaperOvO/AdaptiveLog-review), locate the uncertainty-estimation loop and verify whether the condition that increments alpha is 'obs <= variation' or 'obs >= variation'. Then re-run the BGL anomaly-detection configuration with the opposite comparison while keeping everything else fixed. If the hard-sample percentage or final F1 changes materially from the reported ~27.7% hard samples, the published routing and cost figures depend on the text/code discrepancy and must be corrected before the central claim can be accepted; if the implemented condition is already 'obs >= variation', the paper text is a sign typo and the core mechanism may survive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The weakest link is the uncertainty count in Section 3.3. Equation (7) defines Variation as the mean absolute deviation of the SLM's Monte-Carlo-dropout probabilities on correct validation samples, and Equation (8) defines Observation_n as the analogous deviation on the new input. The text then sets alpha = #{n : Observation_n <= Variation}. Correct samples are stable, so Variation is the small-variation baseline; a hard, error-prone sample should show Observation_n > Variation on most passes. Counting the opposite inequality therefore assigns low alpha, hence low p(C|(x,y')), to exactly the samples that should be routed to the LLM, and high p(C) to easy samples, inverting the selection rule in Equation (9). This is not a cosmetic typo: Tables 3-5 show that the samples selected as hard have much lower SLM accuracy (BGL hard-sample F1 is 0.0 versus 99.84 on easy samples), which can only happen if those samples receive high alpha under the implemented code. The text and the experimental behavior cannot both be as written. Section 5.1 repeats the inversion by saying models are 'prone to errors when their uncertainty probability is low' immediately after reporting that error samples cluster at high uncertainty. The 73% cost saving and the superior-results claim both collapse if the routing is actually inverted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaptiveLog, a hybrid log-analysis framework that first obtains a prediction from a fine-tuned small language model (SLM), estimates the SLM's uncertainty via Monte Carlo dropout and Bayesian updating, and invokes a large language model (LLM) only when the estimated uncertainty exceeds certainty. For LLM-routed samples, the framework retrieves similar error-prone cases from a validation-derived mistake database and prompts the LLM with their reasoning processes and potential pitfalls. Experiments are reported on six log-analysis tasks (anomaly detection, failure identification, module classification, level prediction, log-description semantic matching, and log-cause ranking), including low-resource and transfer settings, with a claimed 73% reduction in LLM query cost relative to querying the LLM on all samples.","tokens_in":30966,"tokens_out":6098,"duration_ms":69389,"significance":"If the central claims hold, AdaptiveLog would be a practically useful contribution: it combines the efficiency of a fine-tuned SLM with the reasoning ability of an LLM, and the idea of retrieving error-prone cases to prompt the LLM is plausible and well motivated. The paper is also commendable for its breadth of evaluation across six tasks, its explicit ablation studies of both the routing mechanism and the prompt strategy, and its provision of source code and detailed experimental data. However, the manuscript as written contains an internal inconsistency in the uncertainty-counting rule that is load-bearing for the headline cost-accuracy claim, and the baseline comparisons are weakened by a potentially unfair re-pre-training procedure for Biglog and KnowLog. These issues must be resolved before the empirical conclusions can be accepted.","major_comments":[{"comment":"The uncertainty-counting rule is inverted as written. The manuscript defines Variation as the mean absolute deviation of Monte-Carlo-dropout probabilities on correct validation samples and then sets alpha = #{n : Observation_n <= Variation}. Because correct samples are stable, Variation is small, and an unstable or error-prone input should exhibit Observation_n > Variation on most passes. The written inequality therefore assigns high alpha to confident samples and low alpha to uncertain ones, inverting the routing decision in Eq. (9). This contradicts the paper's own partition: Table 3 labels hard samples as high-uncertainty and reports BERT hard-sample F1 of 0.0 on BGL while easy samples reach 99.84, a separation that can only arise if the implemented code counts the opposite inequality. Section 5.1 further states that models are 'prone to errors when their uncertainty probability is low' immediately after reporting that error samples cluster at uncertainty 0.8–1.0. The authors must correct the inequality, or explicitly explain the intended semantics, and reconcile these statements; as written, the 73% cost-saving and superior-results claims are unsupported.","section":"§3.3, Eq. (8) and following text"},{"comment":"Equation (4) writes the likelihood as p((x,y')|C) ≈ p(C)^alpha (1-p(C))^{N-alpha}, substituting the prior probability for the Bernoulli parameter. The likelihood of observing alpha uncertain outcomes given parameter theta should be theta^alpha (1-theta)^{N-alpha}, and the posterior in Eq. (5) is indeed the correct conjugate update for that likelihood. As written, however, Eq. (4) conflates the prior with the likelihood parameter and is mathematically inconsistent. In addition, setting p(C) = err from Eq. (1) assumes that the unobservable 'uncertain' event C coincides with classification error; this equivalence is asserted rather than justified. The final update formula in Eq. (6) is standard, but the derivation needs to be rewritten and the assumption stated explicitly.","section":"§3.3, Eqs. (4)–(6)"},{"comment":"The baseline comparison for Biglog and KnowLog is potentially unfair as described. The text says these models are 'reproduced with the same pre-training setting on our log corpus (all training sets),' and the training sets are very small (e.g., 196 samples for OpenStack and roughly 400–7,000 samples for the network-device tasks). The documented advantage of Biglog and KnowLog comes from large-scale log pre-training, so re-pre-training them from scratch on the task corpora likely removes their main strength and makes the 'state-of-the-art' claim less convincing. Please clarify whether the original pre-trained checkpoints were used or whether the models were trained from scratch on the small corpora, and if the latter, add comparisons with the original pre-trained versions or justify why this setting is appropriate.","section":"§4.3, Tables 3–5"}],"minor_comments":[{"comment":"The text references 'Table ??' when discussing the effect of the proportion of error samples on ECR; this should point to Fig. 14 or a properly numbered table.","section":"§5.1, Table 9 and Fig. 14"},{"comment":"The threats-to-validity paragraph on randomness says 'Randomness in the selection of error cases in RQ4,' but the random-case experiment appears in RQ6 (Fig. 10), not in the RQ4 study of different SLMs; the cross-reference should be corrected.","section":"§5.3"},{"comment":"Several entries in Tables 3–5 have formatting problems, such as '86.0 /96.0' and similarly missing spaces around slashes; these should be cleaned up for readability.","section":"Tables 3–5"},{"comment":"The phrase 'the first term is a constant' is imprecise: err/(N+1) is constant with respect to the observation count alpha, but it depends on the validation error rate; this should be stated more carefully.","section":"§3.3, Eq. (6)"},{"comment":"The paper calls the approach 'simple and non-parametric,' but several hyperparameters are set empirically, including N (number of dropout observations), the routing threshold of 0.5 in Eq. (9), k (number of retrieved cases), and the similarity threshold for case quality; the 'non-parametric' phrasing should be softened or qualified.","section":"§1 and §3.4"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the experimental scope is unusually broad for this area, with useful ablations and a public repository. The main obstacle is the internally contradictory description of the uncertainty-counting rule in §3.3, which, taken literally, would invert the selection mechanism; because the reported easy/hard partitions behave as if the correct inequality were implemented, this is likely a writing error rather than a fundamental flaw, but the authors must fix it and reconcile the surrounding text before the cost-accuracy claims can be evaluated. The baseline fairness issue for Biglog and KnowLog also needs a clear response, as it affects the strength of the state-of-the-art claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my read of AdaptiveLog. The genuinely new piece is the ECR prompt: retrieving the SLM's past errors with LLM-generated reasoning and pitfalls to help the LLM on routed hard samples. That is concrete, and the ablations support it—ECR beats ICL on hard samples in most tasks, and removing the Reason field hurts. The uncertainty-based cascade is not conceptually new; model cascades are standard. But applying it to log analysis with a validation-derived error rate as the prior is a reasonable engineering choice, and the evaluation spans six tasks, low-resource and transfer settings, and several SLM backbones. They also ship code and data, which I value.\n\nThe load-bearing soft spot is the routing rule in §3.3. As written, Equation (8) counts Observation_n <= Variation as the uncertain event. Correct validation samples are stable, so Variation is the small-deviation baseline; a hard sample should show Observation_n > Variation. Counting the wrong inequality would route easy samples to the LLM and leave hard samples with the SLM. The tables show the opposite pattern, so either the equation is a typo and the code uses \">\", or the experimental results do not follow from the stated method. Section 5.1 adds to the confusion by saying models are \"prone to errors when their uncertainty probability is low\" right after showing that error samples cluster between 0.8 and 1.0. This must be resolved by pointing to the code or by re-running.\n\nOther soft spots, in proportion. The Bayesian derivation in Equation (4) treats p(C) as both prior and likelihood; the final posterior mean is a defensible shrinkage estimator, but the derivation needs rewriting. Biglog and KnowLog are re-pretrained on this paper's small training sets, which is not the same regime as their original large-scale pretraining, so calling them strong baselines is generous. The main tables have no error bars or multiple seeds; the text mentions averaging over three selections only for one RQ. There is also a missing \"Table ??\" reference in Section 5.1, plus assorted typos. All fixable.\n\nVerdict: this deserves a serious referee. The ECR contribution and the breadth of evaluation justify the time. Acceptance should be conditional on resolving the sign inconsistency with code evidence, because the 73% cost saving and the \"superior results\" claim collapse if the implemented routing is actually inverted. If the code confirms the intended inequality, the central claim is plausible and the paper is a useful contribution to applied log analysis.","headline":"A practical SLM/LLM cascade for log analysis with a genuinely useful error-case retrieval prompt, but the uncertainty routing rule in §3.3 is inverted as written and must be resolved before the 73% cost claim can be trusted.","tokens_in":31480,"tokens_out":2547,"would_cite":false,"duration_ms":30386,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaptiveLog claims that routing only uncertain logs to a large language model beats full-LLM analysis at 27% of the LLM cost.","keywords":["adaptive log analysis","large language model","small language model","uncertainty estimation","Monte Carlo dropout","in-context learning","case-based reasoning","cost-efficient inference"],"falsifier":"Take any held-out test set from the paper (for example, LDSM Huawei-Switches), run the SLM with ten dropout passes per sample, and compare the estimated uncertainty probability with whether the SLM's prediction is actually wrong; if uncertainty is not systematically higher for misclassified logs than for correct logs, or if the Section 3.3 rule selects the opposite subset, the routing mechanism is not doing the work and the framework's gains would disappear.","tokens_in":30510,"feed_emoji":"🤖","tokens_out":9367,"duration_ms":92924,"temperature":0.7,"pith_summary":"AdaptiveLog tries to prove that automated log analysis can get large-language-model accuracy at a fraction of the cost by letting a cheap, fine-tuned small model answer most logs and invoking an expensive LLM only when the small model is uncertain. The paper also claims that the LLM's hard-case answers improve when its prompt includes retrieved examples of the small model's past mistakes, together with reasoning and pitfalls. If these claims hold, operators of large systems could cut LLM inference expense by about 73% while improving overall accuracy, and the same collaboration recipe could be reused for other tasks with a cheap model and an expensive one.","feed_headline":"Log-analysis framework cuts LLM cost by 73% with higher accuracy","feed_subtitle":"Routine logs stay on a cheap fine-tuned model; the expensive LLM is invoked only for hard cases.","key_machinery":"The central mechanism is the uncertainty estimate $p(C|(x,y'))$ computed after the SLM predicts: the SLM is run $N=10$ times with dropout, the deviation of each run from the mean is compared with the variation seen on correct validation samples, and the count of 'uncertain' observations updates a Beta prior $\\mathrm{Beta}(err,1-err)$ to a posterior whose expectation is the routing score. The LLM is invoked when that score exceeds 0.5. The second mechanism is the Error-Case Reasoning Enhancement (ECR), a prompt strategy that retrieves similar error-prone cases: validation errors of the SLM are converted by the LLM into cases containing reasoning and pitfalls, stored in a key-value database, retrieved by cosine similarity using an embedding model, and placed in the prompt in ascending similarity order to counter recency bias.","core_discovery":"The paper's central claim is that an adaptive router can give a large language model's accuracy at a small model's cost: on six log-analysis tasks (anomaly detection, failure identification, module classification, level prediction, log-description semantic matching, and log-cause ranking), AdaptiveLog outperforms both the fine-tuned BERT small model and a ChatGPT baseline that analyzes every sample, while invoking the LLM for only about 27% of logs on average. The router uses a Bayesian posterior over the small model's error probability, updated by repeated Monte Carlo dropout passes, to decide when the LLM is needed; the LLM's hard-sample answers are then improved by the ECR prompt, which retrieves similar past mistakes of the small model along with their reasoning and pitfalls. The paper reports that ECR beats standard in-context learning on hard samples and that the same recipe improves results across different SLMs, low-resource settings, and cross-vendor transfer.","pith_inferences":["Beyond logs, the same two-model recipe should transfer to any high-volume classification task where a cheap fine-tuned model handles most inputs and an expensive model is reserved for the uncertain tail; the paper does not test that, but nothing in the method is log-specific.","The reported 73% saving is measured against a ChatGPT baseline that analyzes every sample; a fairer operational comparison would also count the GPU time of the SLM's ten dropout passes, which the paper reports as about 4.68% of runtime but does not fold into the dollar cost.","A natural stress test is to replace the Monte Carlo dropout variation rule with a calibrated confidence score (softmax entropy or a temperature-scaled probability) and check whether the same cost-accuracy frontier holds; the paper's own learned-router ablation suggests routing is hard, so a calibrated simple baseline would be informative."],"forward_implications":["Operators can cut LLM API spend by about 73% on similar workloads while keeping or improving accuracy, because only about 27% of logs on average are routed to the LLM.","In low-resource settings with 10% of the training labels, the framework's advantage over all baselines grows, since the weaker SLM sends more samples to the LLM.","The framework is not tied to BERT: combining it with Biglog, KnowLog, Llama-3.2-3B, or ChatGLM-6B improves hard-sample accuracy over each base model alone.","Prompting the LLM with similar error-prone cases that include a reasoning process and potential pitfalls beats standard in-context learning and random error cases on hard samples.","The SLM uncertainty estimation is cheap enough (a few percent of total runtime) that the framework remains faster overall than analyzing every log with ChatGPT."],"supporting_citations":[{"why":"Supplies the fine-tuned BERT SLM that handles easy logs and exposes uncertainty.","marker":"[10]"},{"why":"Provides Monte Carlo dropout, the repeated forward passes with random neuron deactivation that give the variation signal used to estimate uncertainty.","marker":"[18]"},{"why":"Provides the conjugate-prior Bayes update used to combine the validation error rate with observed variation into a posterior.","marker":"[29]"},{"why":"Motivates the core assumption that a model's predictions are unstable when they are wrong.","marker":"[31]"},{"why":"Introduces case-based reasoning, the inspiration for retrieving past error cases for the LLM prompt.","marker":"[32]"},{"why":"Reviews case-based reasoning and grounds the ECR prompt strategy.","marker":"[62]"},{"why":"Demonstrates recency bias in few-shot prompting, motivating the ascending-similarity ordering of error cases.","marker":"[75]"},{"why":"Supplies the self-consistency strategy used to check and filter low-quality error-prone cases in the database.","marker":"[61]"},{"why":"Provides the BGL and ThunderBird anomaly-detection datasets and the windowing protocol used in evaluation.","marker":"[35]"}],"fun_headline_variants":["Adaptive router: LLM accuracy at small model cost","Log analysis: LLM on tough cases, SLM on easy","AdaptiveLog: know when to ask the LLM","LLM cost down 73% in log analysis with adaptive routing","Small model routes logs, big model handles hard cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Section 3.3 counting rule, which treats a small deviation between repeated predictions as an uncertain event, correctly identifies logs the small model will get wrong; if that sign convention is wrong, the router sends the wrong samples to the LLM and the claimed cost-accuracy trade-off collapses.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive router: LLM accuracy at small model cost","Log analysis: LLM on tough cases, SLM on easy","AdaptiveLog: know when to ask the LLM","LLM cost down 73% in log analysis with adaptive routing","Small model routes logs, big model handles hard cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1565,"prompt_tokens":1059,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":422}},"tokens_in":675,"tokens_out":506,"duration_ms":5613,"temperature":1.0,"reasoning_tokens":422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:43:14.892743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any held-out test set from the paper (for example, LDSM Huawei-Switches), run the SLM with ten dropout passes per sample, and compare the estimated uncertainty probability with whether the SLM's prediction is actually wrong; if uncertainty is not systematically higher for misclassified logs than for correct logs, or if the Section 3.3 rule selects the opposite subset, the routing mechanism is not doing the work and the framework's gains would disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Monte Carlo dropout, the repeated forward passes with random neuron deactivation that give the variation signal used to estimate uncertainty."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the conjugate-prior Bayes update used to combine the validation error rate with observed variation into a posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces case-based reasoning, the inspiration for retrieving past error cases for the LLM prompt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reviews case-based reasoning and grounds the ECR prompt strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates recency bias in few-shot prompting, motivating the ascending-similarity ordering of error cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-consistency strategy used to check and filter low-quality error-prone cases in the database."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BGL and ThunderBird anomaly-detection datasets and the windowing protocol used in evaluation."}],"review_version":1}