{"id":"5cf94be3-842f-48e7-8eb9-9eb478ef5f89","arxiv_id":"2501.11247","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GAT-LLM, a hybrid of GPT-2 and a graph attention network, reports improved multivariate wireless link quality prediction over four baselines, but lacks a described train/test split and code release.","lead":"The paper proposes GAT-LLM, a model combining graph attention networks with a fine-tuned GPT-2 language model to predict wireless link quality from nine protocol-layer measurements. On a public China Mobile dataset it reports lower prediction error than four baselines, especially for multi-step forecasts, though the experimental details are incomplete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split or normalization-fit details are reported, so the reported MAE/RMSE may reflect training-set memorization and test leakage; the central outperformance claim is unverified.","rationale":"The paper's sole evidence for GAT-LLM's superiority is the MAE/RMSE table and multi-step curves. If the evaluation leaks information, the entire comparison is meaningless. This is the most load-bearing assumption because every other claimed advantage (cross-variable GAT, multi-step gain) is expressed through these numbers. The reader identified this as the weakest assumption, and I concur. I also note secondary inconsistencies, such as Table II listing M=4 output parameters while Table III reports predictions for 9 variables, and the absence of error bars, but these are subordinate to the missing evaluation protocol. The proposed test—a temporal split with train-only normalization—would settle the concern. If the gaps persist on a held-out test, the claim stands; if they do not, the central contribution collapses. Therefore, the CONDITIONAL verdict remains appropriate: the paper needs to state and implement a proper evaluation protocol before its claims can be accepted.","tokens_in":8662,"tokens_out":5587,"duration_ms":53013,"concrete_test":"Run a controlled re-evaluation: split the 22,661-sample series temporally (e.g., train on samples 1–18,128, test on 18,129–22,661), fit min-max normalization on the training split only, and recompute Table III and Fig. 3 for GAT-LLM and all baselines. If GAT-LLM's MAE/RMSE on the held-out test set no longer beats GPT-2 and GAT-Transformer on most variables (or the gap is within noise), the central performance claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GAT-LLM 'significantly improves' link quality prediction over GPT-2, GAT-Transformer, Conv-LSTM, and VARIMA rests entirely on the MAE/RMSE values in Table III and Fig. 3. The manuscript never specifies a train/test split: Section IV-A describes only epochs, batch size, optimizer, and learning rate, with no split ratio, no indication that the split is temporal, and no validation set. Given the data are a single 22,661-sample time series with stride-1 sliding windows, a random split would produce train and test windows that overlap by 19/20 samples, leaking test information into training. Moreover, Section III-C states min-max normalization is applied 'to scale all parameters linearly to [0,1]' without saying whether the bounds are fitted on the training set only; if fitted on the full series, future statistics enter the trained model. Because the model is fine-tuned for 500 epochs on (implicitly) the full data, it can memorize the training set, and the reported numbers would then reflect memorization rather than generalization. Without a held-out test set and normalization fitted on training data, the reported performance gaps are not evidence that GAT-LLM generalizes. The paper also reports no error bars or multiple-seed variance, so even with a split the significance of the gaps is unknown. Section IV-B's statement that GAT-LLM outperforms 'across most variables' is thus unverifiable from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GAT-LLM, a model that uses a graph attention network over nine cross-layer wireless link quality variables and feeds the resulting embeddings into a fine-tuned GPT-2 to produce one-step and autoregressive multi-step predictions. The authors evaluate on a public 22,661-sample China Mobile dataset and compare against GPT-2, GAT-Transformer, Conv-LSTM, and VARIMA using MAE and RMSE. The central claim is that GAT-LLM outperforms all baselines across most variables and improves robustness, especially in multi-step prediction scenarios.","tokens_in":8973,"tokens_out":4673,"duration_ms":41878,"significance":"The combination of GAT with a pretrained LLM for multivariate wireless link quality prediction is a timely and plausible idea, and the use of a public dataset plus several baselines is a strength. If the performance advantage were established under a sound evaluation protocol, the paper would make a useful contribution to LLM-based time series forecasting in wireless networks. However, the current experimental description omits essential details about data splitting and normalization, and no uncertainty quantification is provided, so the empirical central claim is not yet supported.","major_comments":[{"comment":"The experimental setup never states how the 22,661-sample series is divided into training and test sets. With the stride-1 sliding windows defined in Eq. (2), a random split would place 19 of every 20 time steps of a test window in the training set, making the reported one-step MAE/RMSE values in Table III uninformative about generalization. Please specify a temporal split (e.g., first 80% train, last 20% test), state the split ratio, and confirm that no test-window time steps appear in training.","section":"IV-A, Table II"},{"comment":"The preprocessing section says only that min-max normalization scales all parameters linearly to [0,1]; it does not state whether the minimum and maximum are computed on the training portion or on the entire series. If the latter, future test statistics enter the trained model through the input embedding, which would inflate the reported performance. Please state that normalization parameters are fitted on the training set only and applied to validation and test sets.","section":"III-C"},{"comment":"The text says GAT-LLM is better 'with two exceptions: DLOccupyPRBNum, which performs worse than the GPT-2 scheme in terms of RMSE,' but only one exception is identified; moreover, Table III shows DLOccupyPRBNum RMSE=0.0154 for GAT-LLM versus 0.0130 for GPT-2. Please correct the count and give a per-variable summary of wins and losses for both MAE and RMSE, since the central outperformance claim is qualified.","section":"IV-B, Table III"},{"comment":"All results appear to come from a single run; no standard deviations, confidence intervals, or multiple-seed experiments are reported. Given that the model fine-tunes all GPT-2 parameters for 500 epochs on 22,661 samples, the observed gaps may be within run-to-run variance. Please report mean plus/minus standard deviation over at least 3 to 5 seeds and, if possible, a paired significance test across variables.","section":"IV-A, Table III, Fig. 3"},{"comment":"The multi-step evaluation protocol is underspecified. Eq. (3) is applied autoregressively, but it is not stated whether the model consumes its own previous predictions or ground-truth values at each step, and the y-axis label in Fig. 3 is 'MAE(dB)' even though several variables (DLMACRate, MCS) are not measured in dB. Please describe the recursive evaluation procedure and use consistent units.","section":"II-B, Fig. 3"},{"comment":"The abstract and conclusion claim improved 'robustness,' but no experiment measures robustness to noise, missing values, or distribution shift; the multi-step curves only show error growth over the horizon. Please either add a robustness evaluation or soften the claim so it matches the experiments actually reported.","section":"Abstract, V"}],"minor_comments":[{"comment":"There are typos such as 'Exising' in the Introduction and 'Bandwith' in Section II-A; please proofread the manuscript.","section":"I"},{"comment":"Eq. (1) writes X_t as a subset of R^{T x N}, but a single multivariate observation should be a vector in R^N; the notation should be corrected. Eq. (2) defines A but never uses it afterward.","section":"II-B, Eq. (1)"},{"comment":"The entries 'Training step length 19' and 'Step size for predicting (l) 10' are not defined in the text; please clarify what these parameters mean.","section":"Table II"},{"comment":"Reference [6] is cited as both 'One fits all' and as the source of GPT-2; the GPT-2 model should be cited to its original authors, not to Zhou et al.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The architecture and problem choice are reasonable, but the experimental protocol is the main blocker: without a described train/test split, normalization fitting procedure, and repeated runs, the reported numbers are not verifiable. I recommend requesting a major revision focused on the evaluation section rather than rejecting outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible architecture paper whose reported results I don't yet trust, mainly because the evaluation protocol is underspecified.\n\nWhat's actually new: applying a GAT over the nine link-quality variables and feeding the resulting token sequence into a fine-tuned GPT-2 for multi-step multivariate forecasting. That combination is new for this domain. The paper is clearly written, the architecture is easy to follow, and the dataset is a real China Mobile trace with proper physical meaning for each variable.\n\nWhat's good: the GAT idea is well motivated. LLMs treat each timestamp as a token and struggle to mix variables, so a graph over variables is a sensible fix. The comparison against GPT-2 alone is a useful ablation showing the GAT part contributes something. The multi-step curves in Fig. 3 look internally consistent, with GAT-LLM leading.\n\nSoft spots, in proportion: the central performance claim rests entirely on MAE/RMSE values, but no train/test split is described anywhere. The data is a single 22,661-sample series with stride-1 sliding windows. If the split is random, about 95% of the training windows overlap test windows, so the \"test\" error could reflect memorization rather than generalization. The min-max normalization in Section III-C says nothing about which set the bounds come from; if fitted on the full series, future statistics leak into training. No error bars or repeated runs are reported, so even with a clean split the significance of the gaps is unknown. The paper also omits the LLM baselines it itself cites (Time-LLM, MultiCast), and Table II says M=4 while the results show nine outputs. These are all fixable, but they make the current version unverifiable.\n\nWho this is for: someone working on wireless link quality prediction or on LLM-based multivariate forecasting will find the architecture interesting and the problem setup useful. A general ML reader would mostly take away a cautionary example of evaluation pitfalls.\n\nRecommendation: send to peer review, but expect heavy revision. The referee should demand a temporal split, normalization fitted on training data only, multiple seeds, and at least one or two of the cited LLM baselines implemented. The idea is worth testing; the current experimental write-up is not yet enough to confirm it.","headline":"Plausible GAT+GPT-2 hybrid for wireless link quality prediction, but the reported results are unverifiable because the train/test split and normalization fit are never specified.","tokens_in":9501,"tokens_out":1884,"would_cite":false,"duration_ms":19278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph attention layer over link parameters lets a fine-tuned GPT-2 predict wireless link quality more accurately than the LLM alone.","keywords":["link quality prediction","large language models","graph attention networks","multivariate time series forecasting","GPT-2 fine-tuning","wireless communications","multi-step prediction"],"falsifier":"Compute the same five-model comparison with a strictly chronological split and with normalization fitted only on the training portion; if GAT-LLM no longer beats GPT-2 alone and GAT-Transformer on most variables, or if its multi-step margins shrink to noise, the paper's central performance claim would be falsified.","tokens_in":8456,"feed_emoji":"📡","tokens_out":7871,"duration_ms":69461,"temperature":0.7,"pith_summary":"GAT-LLM is a proposed architecture for wireless link quality prediction that combines a graph attention network with a fine-tuned pretrained large language model. The paper's central claim is that treating nine protocol-layer link parameters as nodes of a graph, letting GAT exchange information across them, and feeding the enriched sequence into GPT-2 yields more accurate multivariate forecasts than GPT-2 alone, GAT-Transformer, Conv-LSTM, or VARIMA, especially for ten-step-ahead prediction. On a 22,661-sample operator-collected base-station dataset with one-millisecond resolution, the reported MAE and RMSE favor GAT-LLM on nearly every variable, with the only tabulated case where another model wins outright being DLOccupyPRBNum RMSE. The value of the claim, if it holds, is a practical route to short-term cross-layer link forecasting at the base station without hand-crafted feature engineering.","feed_headline":"Graph attention makes GPT-2 a sharper wireless link predictor","feed_subtitle":"GAT-LLM beats GPT-2, GAT-Transformer, Conv-LSTM and VARIMA on a nine-variable base-station radio dataset.","key_machinery":"The load-bearing mechanism is a graph attention layer across variables: nine protocol-layer parameters become nodes, a learnable matrix transforms their features, and a multi-head attention coefficient $\\alpha_{jk}$ aggregates neighbor information before a linear layer turns the enriched vectors into GPT-2 tokens with positional encoding. This is what lets a one-dimensional-sequence LLM see multivariate structure; the paper's argument is that this cross-variable exchange is the reason GAT-LLM beats GPT-2 alone, while the pretrained transformer supplies temporal pattern recognition.","core_discovery":"The core discovery the paper asserts is that the bottleneck for LLM-based link quality prediction is not the LLM's temporal modeling but its blindness to interdependencies among simultaneously measured variables, and that a graph attention layer placed before the LLM removes that bottleneck. Concretely, GAT-LLM constructs one graph per time step with nine nodes (DLBw, ULSINR, DLOccupyPRBNum, CellDLMACRate, DLMACRate, MCS, PDCPOccupyBuffer, PDCPUnusedBuffer, DLPDCPSDUNum), computes multi-head attention coefficients between nodes, aggregates neighbor features, linearly projects the result into LLM tokens, and fine-tunes all GPT-2 parameters on the prediction task. The experiments report lower MAE and RMSE than the four baselines for one-step forecasts across most variables and for all four reported parameters over ten steps, and the multi-step comparison shows the margin grows with horizon, which the paper attributes to GAT's cross-variable information feeding the autoregressive loop.","pith_inferences":["If the reported gains survive a clean chronological split, the same graph-over-variables design could transfer to other multivariate radio-management tasks, such as handover timing or energy-efficiency control, because the GAT layer is agnostic to which protocol parameters form the nodes.","The paper does not run an ablation that fixes GPT-2 and toggles the GAT layer on and off, so the reader cannot yet tell how much of the improvement is due to cross-variable graph attention and how much to fine-tuning the LLM itself; that ablation would be a direct test of the paper's explanatory story.","Because all experiments use one 22,661-sample trace from a single operator at 1 ms resolution, the paper's robustness claim remains a single-dataset observation; validating it on other operators, mobility regimes, and radio environments would show whether the architecture generalizes or overfits to this trace."],"forward_implications":["With GAT-LLM, a base station could turn nine 1 ms link metrics into 10 ms look-ahead forecasts of downlink bandwidth, user rate, MCS, and uplink SINR, providing a short prediction window for scheduling and link adaptation.","The GAT layer removes the need to hand-engineer cross-layer features: the model learns which protocol-layer variables matter to each other directly from data.","The reported multi-step margin over univariate prediction implies that correlated variables contribute most at longer horizons, exactly where autoregressive error accumulation normally worsens forecasts.","The paper's configuration shows a pretrained transformer can be fine-tuned on a relatively small domain dataset (22,661 samples) rather than retrained, which the paper argues is a practical route to LLM-based prediction at the base station."],"supporting_citations":[{"why":"supplies the pretrained GPT-2 backbone that GAT-LLM fine-tunes for time series prediction","marker":"[6]"},{"why":"demonstrates reprogramming a pretrained LLM for time series forecasting, the approach GAT-LLM adapts to link quality","marker":"[7]"},{"why":"defines the GAT-Transformer baseline that combines graph attention with a transformer and must be beaten","marker":"[12]"},{"why":"defines the Conv-LSTM baseline for link quality prediction in millimeter-wave systems","marker":"[3]"},{"why":"defines the (V)ARIMA statistical baseline and the application-level link quality prediction setting","marker":"[2]"},{"why":"documents the one-dimensional limitation of LLMs for multivariate time series, motivating the graph attention component","marker":"[9]"}],"fun_headline_variants":["Graph attention boosts GPT-2 for wireless link forecasting","GAT-LLM weds GPT-2 with graph nets for link quality","Why graph attention makes GPT-2 a top link predictor","Multivariate link prediction: GAT gives GPT-2 the missing context","GPT-2 gets graph vision for sharper wireless link forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance comparison rests on the assumption that the min-max normalization bounds and the train/test split respect temporal order and use only training data; the paper does not state either, so if normalization spans the full 22,661-sample series, test information leaks into training and the reported MAE/RMSE margins would be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Graph attention boosts GPT-2 for wireless link forecasting","GAT-LLM weds GPT-2 with graph nets for link quality","Why graph attention makes GPT-2 a top link predictor","Multivariate link prediction: GAT gives GPT-2 the missing context","GPT-2 gets graph vision for sharper wireless link forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001165,"raw_usage":{"total_tokens":4824,"prompt_tokens":950,"completion_tokens":3874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":3786}},"tokens_in":566,"tokens_out":3874,"duration_ms":23146,"temperature":1.0,"reasoning_tokens":3786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:28:41.659277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same five-model comparison with a strictly chronological split and with normalization fitted only on the training portion; if GAT-LLM no longer beats GPT-2 alone and GAT-Transformer on most variables, or if its multi-step margins shrink to noise, the paper's central performance claim would be falsified.","supporting_citations":[{"cited_title":"Time-LLM: Time Series Forecasting by Reprogramming Large Language Models,","cited_arxiv_id":null,"evidence_quote":"demonstrates reprogramming a pretrained LLM for time series forecasting, the approach GAT-LLM adapts to link quality"},{"cited_title":"Vehicular Crowdsensing Inference and Prediction With Multi Training Graph Transformer Networks,","cited_arxiv_id":null,"evidence_quote":"defines the GAT-Transformer baseline that combines graph attention with a transformer and must be beaten"},{"cited_title":"Point Cloud- Based Proactive Link Quality Prediction for Millimeter-Wave Commu- nications,","cited_arxiv_id":null,"evidence_quote":"defines the Conv-LSTM baseline for link quality prediction in millimeter-wave systems"},{"cited_title":"Long-term application-level wireless link quality prediction,","cited_arxiv_id":null,"evidence_quote":"defines the (V)ARIMA statistical baseline and the application-level link quality prediction setting"},{"cited_title":"MultiCast: Zero- Shot Multivariate Time Series Forecasting Using LLMs,","cited_arxiv_id":null,"evidence_quote":"documents the one-dimensional limitation of LLMs for multivariate time series, motivating the graph attention component"}],"review_version":1}