{"id":"5a096123-9454-4e15-b68a-abd37f114052","arxiv_id":"2502.00022","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning Qwen2.5-7B on reactor-operator simulator data yields workload estimates that the authors report as more accurate than zero-shot commercial LLMs, but the evaluation lacks a demonstrated train/test split.","lead":"This paper fine-tunes a large language model on simulator data from nuclear reactor operators to estimate how much mental workload each operator role carries in different scenarios. The authors claim the resulting tool, WELLA, predicts NASA-TLX workload scores more accurately than commercial chatbots like GPT-4 and Claude.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split is reported, so the high R2 may reflect memorization; Table 6 is identical to Table 5, independently indicating unreliable evaluation. The central 'prediction accuracy' claim is unsupported until held-out evaluation is shown.","rationale":"I read the paper as claiming that a fine-tuned LLM can generate synthetic HRA workload data with accuracy close to human operator reports. The reader identified the missing train/test split as the weakest assumption. This is correct and load-bearing: without disjoint evaluation, every result in Tables 1–6 is compatible with memorization. I add that Table 6 duplicating Table 5 is objective evidence that the numerical results as printed cannot be fully trusted; the aggregate row should differ from a single role. The novelty of applying SFT to workload estimation is plausible, but the current manuscript does not support it. I recommend no change to the reader's REJECT verdict: the manuscript needs a revised evaluation protocol, release of data and code, and corrected tables and references before it can be assessed.","tokens_in":12721,"tokens_out":4090,"duration_ms":41509,"concrete_test":"Re-run the evaluation with a scenario-level holdout: train on Startup and Shutdown sub-scenarios only, evaluate on all Accident instances, and report the exact number of SFT training examples that overlap with the 69 evaluation instances. If held-out R2/RMSE/MAE for RO1–CO remain close to Table 1 values (R2≈0.90), the concern is refuted; if R2 collapses toward zero or negative, the reported numbers are in-sample and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—WELLA predicts workload on new scenarios better than commercial LLMs—requires that the R2/RMSE/MAE/EV reported in Tables 1–6 measure out-of-sample generalization. The paper never supplies that guarantee. Section 3.3 builds the virtual cognitive trajectory library from real operator data via Claude; Section 3.4 fine-tunes Qwen2.5-7B on that library; Section 4.2 evaluates WELLA on the same scenario suite (Startup, Shutdown, Accident with 28+11+30 instances). No train/validation/test split, no scenario-level holdout, and no overlap audit between SFT examples and evaluation samples are reported. If the 69 instances (or their LLM-generated trajectories) appear in both the fine-tuning set and the evaluation set, then R2≈0.90 in RO1/RO2/RO3 is exactly what a fine-tuned model memorizing labels would produce, and the abstract's 'prediction accuracy' claim collapses. Independent evidence that the evaluation block is unreliable: Table 6 ('ALL data') is numerically identical to Table 5 ('SO'), which cannot be correct for an aggregated dataset unless ALL=SO; this alone invalidates the aggregate claim. The comparison against zero-shot GPT-4/4o/Claude-3.5 is also not an apples-to-apples test, but the decisive gap is the missing held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WELLA, a fine-tuned Qwen2.5-7B model that estimates operator workload (NASA-TLX) and situational awareness (SART-derived SA) in a multi-agent HTGR main-control-room setting. The pipeline consists of collecting real operator workload data on a simulator testbed, using Claude to generate 'virtual cognitive trajectories' based on macrocognitive theory, fine-tuning Qwen2.5-7B on that trajectory library via Llama-Factory, and then comparing WELLA against zero-shot GPT-4, GPT-4o, and Claude-3.5-Sonnet on per-role regression metrics (R², RMSE, MAE, EV). The authors report that WELLA outperforms the commercial baselines, particularly for RO1/RO2/RO3 (R² 0.90–0.96), with weaker results for CO and SO (R² 0.42 and 0.38). The central claim is that this constitutes a high-precision, dynamic, scenario-driven method for synthetic HRA workload data collection.","tokens_in":13019,"tokens_out":5660,"duration_ms":54752,"significance":"The motivation is solid: HRA data scarcity and the static nature of survey-based workload measurements are real problems, and the use of a real HTGR simulator testbed with actual operators is a strength. The idea of grounding fine-tuning data in macrocognitive trajectories is also interesting and could be valuable for dynamic HRA. However, the current evaluation does not establish out-of-sample prediction accuracy: no train/test split is reported, the commercial baselines are not fine-tuned, and Table 6 is numerically identical to Table 5, which invalidates the aggregate comparison. If a proper held-out evaluation were provided, the approach could be relevant to HRA and human-factors communities, but the evidence in this version does not support the paper's headline claim. No code or data are provided to aid reproducibility.","major_comments":[{"comment":"The paper does not report any train/validation/test split: Section 3.4 fine-tunes Qwen2.5-7B on a cognitive-trajectory library constructed in Section 3.3 from the same real operator workload data collected in Section 3.2, and Section 4.2 evaluates on the same scenario suite (Startup, Shutdown, Accident with 28, 11, and 30 instances). Without a stated held-out set or an overlap audit, the high R² values in Tables 1–5 can be explained by the model memorizing training labels, and the abstract's claim of 'prediction accuracy' is unsupported. A scenario-level holdout or an explicit non-overlap analysis is required.","section":"§3.3–3.4, §4.2"},{"comment":"Table 6 (ALL data) is numerically identical to Table 5 (SO) for every row and every metric, even though the text describes Table 6 as the combined dataset. This is impossible for a proper aggregation unless the ALL dataset coincides with the SO subset; the discrepancy invalidates the aggregate comparison and must be corrected with the actual combined metrics.","section":"Table 6 vs Table 5"},{"comment":"The evaluation compares fine-tuned WELLA with zero-shot commercial models (GPT-4, GPT-4o, Claude-3.5-Sonnet). This is not an apples-to-apples comparison: any advantage could come simply from exposure to real operator labels during fine-tuning. The authors should add few-shot baselines for the commercial models or a same-data fine-tuned open-source baseline to support the claim of superior prediction accuracy.","section":"§4.2, Tables 1–6"},{"comment":"The evaluation reports point estimates without error bars, confidence intervals, or significance tests, and the per-role sample sizes are small (11–30). For example, the CO and SO results (R² 0.42 and 0.38) might not be distinguishable from the commercial baselines under resampling. Bootstrap intervals or paired tests should be reported before drawing comparative conclusions.","section":"§4.2, Tables 1–6"}],"minor_comments":[{"comment":"The section title 'Vitural cognitive trajectory Generation' contains a typo: 'Vitural' should be 'Virtual'.","section":"§3.3"},{"comment":"Equation references point to 'equation 3.2' before the equations are numbered; the equations should be numbered and referenced consistently.","section":"§3.2"},{"comment":"The text says 'Detailed implementation can be found in ??'; this unresolved cross-reference should be replaced with an actual appendix or hyperparameter table.","section":"§4.1"},{"comment":"The model is referred to as 'VELLA' in Section 4.4 and Figure 7; this should be 'WELLA'.","section":"§4.4, Figure 7"},{"comment":"The sentence 'The evaluation is conducted for five scenarios: RO1, RO2, RO3, CO, SO, and the aggregated dataset (ALL)' lists six items; rephrase to clarify the five per-role scenarios plus the pooled dataset.","section":"§4.2"},{"comment":"The SART items are said to be detailed in Appendix D, but no appendix appears in the manuscript; provide the appendix or remove the reference.","section":"§3.2"},{"comment":"The keywords contain a double comma in 'generation method,, Personnel Workload', and the phrase 'currently unavailable commercial models' in Section 4.2 should be reworded.","section":"Keywords and §4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope, but the evaluation issues are serious enough that the current version should not be published as is. The missing train/test split and the identical Tables 5 and 6 are load-bearing problems, though they are potentially fixable with a re-analysis or new experiments. I would also note that several self-citations in the reference list (e.g., refs [1], [48], [51], [52]) are tangential to the methodological claims and could be trimmed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. Short version: the idea is legitimate and the domain is worthwhile, but the evaluation as written does not support the central claim.\n\nWhat's actually new: they fine-tune Qwen2.5-7B on NASA-TLX/SART workload ratings collected from real HTGR main control room operators across startup, shutdown, and accident scenarios, and compare against zero-shot GPT-4/4o and Claude-3.5-Sonnet. The five-role multi-agent framing and the use of LLM-generated 'cognitive trajectories' as an intermediate representation is a reasonable way to get dynamic workload estimates for HRA. If done carefully, that could be a useful, cheaper source of synthetic workload data. I think the reader's significance_if_true of 4 is about right.\n\nWhat's missing: the paper never reports a train/test split. Section 3.3 builds a trajectory library from real operator data, Section 3.4 fine-tunes on it, and Section 4.2 evaluates on the same scenario suite. If the fine-tuning examples overlap with the evaluation instances, then R2 around 0.9 is memorization, not prediction. The stress-test is right that this is the load-bearing issue, and I don't see anything in the text that rules it out. Independent evidence of sloppiness: Table 6 ('ALL data') is numerically identical to Table 5 ('SO'), which cannot be right unless the aggregated set is just SO. Either way the aggregate comparison is invalid. There are also small things: 'Detailed implementation can be found in ??' is an unresolved placeholder, and the commercial baselines are zero-shot, which is a weak comparison for a fine-tuned model.\n\nCredit where due: the paper does not hide some limitations—Table 7 explicitly says WELLA's generalizability is 'No,' and the SO results are acknowledged as weak. But the abstract overstates the result by claiming 'prediction accuracy' without a held-out evaluation.\n\nWho this is for: people working on HRA data synthesis and LLM-based workload estimation. The paper is not publishable in this form, but the direction deserves a serious referee. I'd send it to review with the expectation of major revision: add a proper scenario-level held-out set, report per-instance details, fix the tables, and replace the placeholder. If the held-out results hold, this becomes a solid application paper.","headline":"Plausible application with a broken evaluation: no held-out split, a duplicated table, and weak baselines; the core idea is worth a revision, not a rejection of the direction.","tokens_in":13576,"tokens_out":2513,"would_cite":false,"duration_ms":22070,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fine-tuned LLM can predict the workload of individual control-room roles more accurately than GPT-4, GPT-4o, and Claude-3.5-Sonnet, making automated scenario-driven synthetic HRA data collection feasible.","keywords":["Dynamic Workload Prediction","Large Language Models","NASA TLX","SART","synthetic HRA data generation method","Personnel Workload","Situational Awareness","Scenario-Driven Assessment"],"falsifier":"Hold out a random set of scenarios—say 20% of the 69 collected—before building the cognitive trajectory library, fine-tune WELLA only on the remaining scenarios, and compute $R^2$, RMSE, MAE, and explained variance on the held-out set. If the held-out $R^2$ drops toward zero or negative, the central claim of prediction accuracy is refuted and the reported $R^2 \\approx 0.90$ is attributable to memorization of the training scenarios.","tokens_in":12499,"feed_emoji":"🧠","tokens_out":7441,"duration_ms":94742,"temperature":0.7,"pith_summary":"This paper tries to establish that supervised fine-tuning of an open-weight language model on real operator survey data and LLM-generated cognitive reasoning chains produces workload estimates accurate enough to serve as synthetic human-reliability-analysis data. The resulting system, WELLA, is evaluated against three commercial LLMs on five operator roles in a high-temperature gas-cooled reactor control room, and the paper reports that WELLA is the only model with positive $R^2$ and explained-variance values across scenarios. If the claim is right, workload labels for new collaborative scenarios could be collected automatically, without expert input or post-hoc questionnaires, addressing a long-standing data-scarcity problem in human reliability analysis. The practical payoff would be a dynamic, role-specific, and auditable source of workload and situational-awareness data for safety training and system design.","feed_headline":"Fine-tuned LLM beats commercial models at workload estimates","feed_subtitle":"WELLA, trained on real reactor-operator data, generates synthetic HRA workloads for control-room teams.","key_machinery":"The load-bearing mechanism is the fine-tuned language model that maps a scenario description to workload and situational-awareness scores through an intermediate 'virtual cognitive trajectory.' Real control-room operators first fill out NASA-TLX (a six-dimension subjective workload scale) and SART (a ten-item situational-awareness rating scale) after simulator runs; the commercial model Claude is then used to rewrite those responses as macro-cognitive reasoning chains; and the open-weight model Qwen2.5-7B is fine-tuned on that library with the Llama-factory toolkit. At inference time, five role-specific agents (SO, RO1, RO2, RO3, CO) read a new scenario and emit a cognitive trajectory plus the questionnaire scores, which is what makes the workload estimate dynamic, role-specific, and inspectable.","core_discovery":"The paper's central claim is that a fine-tuned Qwen2.5-7B model—trained on a 'virtual cognitive trajectory library' built by having Claude expand real operators' NASA-TLX and SART responses into five-stage macro-cognitive narratives (detection, understanding, decision-making, action execution, inter-team coordination)—can predict operator workload from scenario text more accurately than GPT-4, GPT-4o, and Claude-3.5-Sonnet. In the reported experiments, WELLA achieves $R^2 = 0.9012$, $0.9343$, and $0.9628$ for the three reactor operators, with lower RMSE and MAE than all baselines; the commercial models mostly show negative $R^2$, meaning their predictions are worse than simply predicting the mean. The paper further claims that this method is fully automated, captures dynamic changes in workload across scenarios, and avoids the expert-input burden of the existing HUNTER simulator, thereby opening a new paradigm for scenario-based synthetic HRA data collection.","pith_inferences":["The paper's own comparison table marks WELLA as 'No' on generalizability, which is at odds with the abstract's claim of adapting to 'various collaborative scenarios'; the most plausible reading is that the high accuracy reflects overlap between the fine-tuning library and the evaluation scenarios rather than transfer to unseen situations.","A decisive extension would be to fine-tune on a subset of roles or reactor modules and predict the held-out roles, testing whether the cognitive-trajectory representation generalizes across job functions rather than merely memorizing per-role phrasing.","The comparison against commercial LLMs as zero-shot baselines necessarily favours the fine-tuned model; an apples-to-apples baseline would be an open-weight model fine-tuned on the same library without the trajectory augmentation, isolating the contribution of the macro-cognitive intermediate step."],"forward_implications":["If the central claim holds, workload labels for new high-temperature gas-cooled reactor scenarios can be generated automatically from scenario text, removing the need for expert knowledge and post-hoc surveys.","The same training recipe—survey labels, LLM-generated cognitive trajectories, and supervised fine-tuning—could be transplanted to other multi-agent high-risk domains such as aviation, healthcare, or chemical plants.","Because the model emits an intermediate cognitive trajectory alongside each score, the resulting synthetic HRA data carries a transparent reasoning record that can be audited for plausibility.","WELLA's additional situational-awareness output gives synthetic HRA datasets a dynamic indicator of risk perception, not just a static workload number."],"supporting_citations":[{"why":"supplies the NASA-TLX ground-truth workload labels used to build the training library","marker":"[38]"},{"why":"supplies the SART situational-awareness ratings used as an additional training target","marker":"[39]"},{"why":"provides the Qwen2.5-7B base model that WELLA fine-tunes","marker":"[43]"},{"why":"provides the Llama-factory framework used for the supervised fine-tuning","marker":"[44]"},{"why":"serves as the GPT-4 commercial baseline that WELLA is compared against","marker":"[46]"},{"why":"serves as the GPT-4o commercial baseline that WELLA is compared against","marker":"[47]"},{"why":"serves as the Claude-3.5-Sonnet commercial baseline that WELLA is compared against","marker":"[48]"},{"why":"defines HUNTER, the existing procedurally driven synthetic HRA method that WELLA is contrasted with","marker":"[49]"}],"fun_headline_variants":["Fine-tuned LLM beats GPT-4 on workload estimates","WELLA predicts operator workload with R-squared above 0.9","LLM trained on reactor data outperforms commercial models","Synthetic HRA data collection via fine-tuned LLM wins","Small fine-tuned LLM surpasses big commercial AI for workload"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy metrics describe prediction of scenarios not seen during fine-tuning; the paper never reports a train/test split, so if the evaluation scenarios overlap with the SFT training library, the high $R^2$ may be memorization rather than prediction.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM beats GPT-4 on workload estimates","WELLA predicts operator workload with R-squared above 0.9","LLM trained on reactor data outperforms commercial models","Synthetic HRA data collection via fine-tuned LLM wins","Small fine-tuned LLM surpasses big commercial AI for workload"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1395,"prompt_tokens":943,"completion_tokens":452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":559,"tokens_out":452,"duration_ms":5094,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:03:07.423267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a random set of scenarios—say 20% of the 69 collected—before building the cognitive trajectory library, fine-tune WELLA only on the remaining scenarios, and compute $R^2$, RMSE, MAE, and explained variance on the held-out set. If the held-out $R^2$ drops toward zero or negative, the central claim of prediction accuracy is refuted and the reported $R^2 \\approx 0.90$ is attributable to memorization of the training scenarios.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the NASA-TLX ground-truth workload labels used to build the training library"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the SART situational-awareness ratings used as an additional training target"},{"cited_title":"Shahriar, B","cited_arxiv_id":null,"evidence_quote":"serves as the GPT-4o commercial baseline that WELLA is compared against"},{"cited_title":"A Hybrid Real-Time Framework for Efficient Fussell-Vesely Importance Evaluation Using Virtual Fault Trees and Graph Neural Networks","cited_arxiv_id":"2412.10484","evidence_quote":"serves as the Claude-3.5-Sonnet commercial baseline that WELLA is compared against"},{"cited_title":"Ulrich, R","cited_arxiv_id":null,"evidence_quote":"defines HUNTER, the existing procedurally driven synthetic HRA method that WELLA is contrasted with"}],"review_version":1}