REVIEW 4 major objections 6 minor 1 cited by
DeepForm: Reasoning Large Language Model for Communication System Formulation
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read DeepForm shows that a 7-billion-parameter model trained on distilled reasoning traces and rule-based reinforcement learning can outperform a 671-billion-parameter reasoning model at writing correct mathematical formulations of…
desk verdict Useful dataset and training recipe, but the headline accuracy claim is unverifiable as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is two-stage training on a purpose-built corpus. In stage one, DeepForm is fine-tuned with LoRA on CSFRC, whose construction pipeline extracts (description, formulation) pairs from the System Model sections of arXiv papers, compresses the descriptions to at most 4096 tokens, and uses DeepSeek R1 with rejection sampling to generate chain-of-thought reasoning paths, with a Levenshtein-similarity-based fallback that completes a path from the correct answer. In stage two, the C-ReMax algorithm, a rule-based variant of ReMax, rewards the model with 1 when the generated formulation is judged equivalent to the reference and 0 otherwise, subtracts a repetition penalty, and reduces gradient variance by contrasting a sampled rollout against a greedy rollout while adding a KL term that anchors the policy to the SFT model. This combination is what the paper credits for eliciting the reasoning behaviors and the accuracy gain.
What would settle it
Compute the overlap between the arXiv papers used for training and those used for testing in CSFRC, then have independent communication engineers write fresh description-formulation pairs from papers published after 2025 and compare DeepForm's accuracy on those against DeepSeek R1; if overlap is substantial or the fresh-set accuracy drops below 68.4%, the reported superiority does not generalize.
Extended reading notes
Core claim
In the paper's own terms, the central discovery is that a 7B model can achieve state-of-the-art performance on communication system formulation by first imitating high-quality reasoning traces and then learning from a deterministic correctness signal. The authors show that supervised fine-tuning alone brings accuracy to 65.1%, and the C-ReMax reinforcement stage raises it to 71.4%, while the base Qwen2.5-7B-Instruct sits at 62.3%. They also report that the trained model develops emergent behaviors such as self-correction, verification, and back-tracking, and they document that giving a negative reward (-1) for wrong answers hurts accuracy by 7.7%, while adding a separate format reward costs another 2%, which they attribute to reduced exploration under negative feedback.
Load-bearing premise
The whole result rests on the assumption that the reference equation extracted from an arXiv paper is the single correct formulation and that the test set, mined from the same 2015-2025 corpus with the same pipeline, is a fair, non-overlapping measure of formulation ability.
Editorial extensions
If this is right
- A 7-billion-parameter model can handle communication system formulation at accuracy above a 671-billion-parameter reasoning model, which lowers the hardware and privacy barrier for using LLMs in this domain.
- The CSFRC dataset, with its description-formulation pairs and CoT annotations, becomes a reusable benchmark and training resource for future work on automated modeling.
- The two-stage recipe — CoT distillation from a large teacher, then rule-based RL with a variance-reduced baseline — is directly transferable to other domains where correct answers can be checked by rule.
- The ablation results imply that adding CoT reasoning traces is not optional: training on question-answer pairs alone degraded the model relative to the base, so data quality matters as much as data scale.
Reading between the lines
- We infer the 'equivalence' reward is likely permissive about algebraic rearrangements and constraint order, so the 71.4% figure may measure equation recognition at a semantic level rather than exact recovery; a human-review study would reveal how strict that match is.
- The same pipeline could be applied to other engineering modeling tasks with extractable paired text-and-equation data, such as circuit analysis or control-system design, where similar knowledge silos exist.
- Because the largest single category of the training set is 'Others' at 37.3%, the reported average accuracy is probably uneven across subfields; a per-category breakdown would show where the method thrives and where it falls back to pattern matching.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DeepForm, a 7B-parameter reasoning LLM for communication system formulation, trained in two stages from a newly constructed corpus CSFRC. The corpus is built from 2015-2025 arXiv papers by parsing "System Model" sections, compressing prompts with DeepSeek V3, and generating chain-of-thought reasoning paths with DeepSeek R1 under rejection sampling and a fallback correction mechanism. Stage one is supervised fine-tuning on the CoT data with LoRA; stage two is a rule-based reinforcement learning algorithm called C-ReMax, based on ReMax with a greedy rollout baseline and KL regularization. The authors report that DeepForm reaches 71.4% accuracy, outperforming DeepSeek R1 (68.4%), GPT-4o Mini, and other baselines, and they attribute the gains to the CoT data and the RL stage.
Significance. If the reported result is valid, it would be a practically interesting demonstration that a 7B open-weights model can beat much larger proprietary and reasoning-focused LLMs on a narrow, domain-specific formulation task. The two-stage training recipe and the C-ReMax reward design are clearly described, and the idea of constructing a communication-system formulation benchmark from arXiv source text is useful for the mobile-computing community. The paper, however, currently provides no publicly available code, dataset, or test set, and the central evaluation is not reproducible from the manuscript alone. The claimed superiority over DeepSeek R1 rests on an undefined equivalence criterion and an unstated test-set construction, which are load-bearing issues for the paper's main conclusion.
major comments (4)
- [Section 4.2, Fig. 7] The central performance claim is not evaluable because the test set is never described: the paper gives no test-set size, no sampling procedure, no prompt template, and no error bars or significance tests. The reported 3.0-point gap (71.4% vs. 68.4%) could easily be within sampling noise on a test set of a few hundred examples. Please report N, exact 95% confidence intervals, and a paired or McNemar test for the comparison against DeepSeek R1.
- [Section 3.1 and Section 4.1] The evaluation shares its source corpus with the training set: CSFRC is built from 2015-2025 arXiv papers in Section 3.1, and Section 4.1 states that all training was performed on CSFRC, with the test set apparently drawn from the same corpus. No deduplication method or paper-level train/test split is reported, so near-duplicate system-model descriptions between training and test could inflate the reported accuracy. Please report the number of distinct arXiv papers in train and test, describe the deduplication procedure (e.g., title, abstract, or embedding overlap), and provide results on a held-out set that is disjoint at the paper level.
- [Eq. (12) and Section 4.2, Eq. (18)] The training reward in Eq. (12) uses "a equivalent to atrue" without defining equivalence, while the reward ablations in Eqs. (18a)-(18b) and the surrounding text use exact equality "a = atrue". These are different evaluation criteria. If the main results in Fig. 7 are scored with a lenient equivalence check, the comparison against DeepSeek R1 is potentially circular, because the same model is used in Section 3.1.3 to judge whether candidate reasoning paths align with the reference formulation. Please specify the equivalence predicate (e.g., normalized string match, symbolic comparison, or an LLM judge), state which model performs the judging, and report accuracies under both exact-match and equivalence-based scoring.
- [Section 3.1.3 and Section 4.2] The benchmark is a self-constructed closed set rather than an external benchmark: reference formulas fj come from the same arXiv parsing pipeline and the same "System Model" sections that generate the training data, and no independent expert verification or manual audit of the test labels is reported. Because the correctness labels themselves may inherit extraction errors or ambiguities, the absolute accuracies are difficult to interpret. Please provide a manually audited or independently verified test subset, report inter-annotator agreement if multiple annotators are used, and state how ambiguous or multi-reference formulations were resolved.
minor comments (6)
- [Abstract and Introduction] There are several typos: "senerios" in the abstract, "equllly" in the footnote, and "Fisrt" in the Conclusion; a careful proofread is needed.
- [Section 2.1] The phrase "dating a general LLM" should read "adapting a general LLM".
- [Eqs. (13)-(16)] The symbol λ is used both for the repetition penalty coefficient in Eq. (14) and for the KL regularization weight in Eq. (16); please rename one of them to avoid confusion.
- [Section 3.3.1] Section 3.3.1 is empty and appears as a dangling heading; either fill it with the RL dataset construction details or remove it.
- [Section 4.2, Fig. 8] The reward-setting ablation is difficult to read: the caption lists three conditions and three bars, but the text describes multiple reward variants; please label the bars explicitly and report the number of test samples and the exact reward settings for each bar.
- [Section 4.2, Fig. 7] The claim that DeepSeek R1 is "94 times larger" is based on total parameters, but DeepSeek R1 is a mixture-of-experts model with far fewer active parameters per token; please state both total and active parameter counts so the size comparison is fair.
Circularity Check
Benchmark self-reference, not equation-level circularity: correctness is defined via a DeepSeek R1-validated pipeline and an undefined equivalence predicate, so the SOTA gap over DeepSeek R1 is not independently grounded.
-
self definitional
[Section 3.1.3 (Eq. 4) and Section 3.3 (Eq. 12), reported in Section 4.2 (Fig. 7)]
"where πR1 represents the generator policy utilizing rejection sampling with the DeepSeek R1 API to evaluate whether a candidate answer aligns with the ground truth ... Ra(a) = (1, if a equivalent to atrue; 0, otherwise) (12)"
The only correctness validator specified in the data pipeline is DeepSeek R1: candidate CoT paths are kept only when R1 says they align with the extracted f_j (Eq. 4). The RL reward in Eq. 12 defines success as 'a equivalent to a_true' without ever defining the equivalence relation or an independent test-set construction. Since DeepForm's SFT supervision is generated and validated by DeepSeek R1, and the Fig. 7 comparison against DeepSeek R1 uses the same undefined correctness notion, the reported 71.4 vs 68.4 gap can measure agreement with the teacher's own equivalence judgment rather than an external standard. This is a partial self-definition of the evaluation metric, not a fitted parameter renamed as a prediction.
full rationale
No equation-level circularity is present in the SFT/RL training losses: Eq. 8 trains on (d_hat, g_tilde, f, label) and Eq. 16 is a standard RL objective; neither reduces to the reported accuracy. The C-ReMax algorithm is a stated variant of external ReMax, and no load-bearing self-citation appears. The main circularity risk is the evaluation: the test set is never described, Eq. 12's 'equivalent' is undefined, and the only named correctness judge in the pipeline is DeepSeek R1, the very model that generated the CoT supervision and the strongest baseline. This makes the central SOTA claim dependent on an internally defined notion of correctness. If the final answers are scored by exact string equality against paper-extracted equations and the test set is deduplicated from training, the result would be largely non-circular; as written, the ambiguity prevents independent verification. Score 3 reflects a mild, evaluation-level circularity, not a derivation that is forced by construction.
Assumptions & free parameters
free parameters (8)
- Lmax prompt length cap =
4096 tokens
- Tmax candidate sampling attempts =
not specified
- Levenshtein similarity thresholds theta and theta2 =
not specified
- repetition penalty lambda =
not specified
- repetition reward cap beta =
not specified
- n-gram range nmin, nmax =
not specified
- KL regularization weight lambda =
0.001
- LoRA rank r =
256
assumptions (4)
- domain assumption Extracting contextual description dj preceding formulation fj from System Model sections yields a complete and unambiguous question-answer pair.
- domain assumption The reference formulation fj extracted from an arXiv paper is the unique correct answer, and DeepSeek R1 can reliably judge whether a candidate matches it.
- standard math The RL objective with greedy-rollout baseline and KL regularization is a valid surrogate for improving formulation accuracy.
- ad hoc to paper Rule-based reward can be computed as exact equivalence to the reference equation.
Cite this review
Pith. "Pith review of DeepForm: Reasoning Large Language Model for Communication System Formulation." pith.science (2026). https://pith.science/paper/MKOUOTZE
@misc{pith2026250608551,
author = {Pith},
title = {Pith review of: DeepForm: Reasoning Large Language Model for Communication System Formulation},
year = {2026},
howpublished = {\url{https://pith.science/paper/MKOUOTZE}},
note = {Machine review of arXiv:2506.08551}
}
read the original abstract
Communication system formulation is critical for advancing 6G and future wireless technologies, yet it remains a complex, expertise-intensive task. While Large Language Models (LLMs) offer potential, existing general-purpose models often lack the specialized domain knowledge, nuanced reasoning capabilities, and access to high-quality, domain-specific training data required for adapting a general LLM into an LLM specially for communication system formulation. To bridge this gap, we introduce DeepForm, the first reasoning LLM specially for automated communication system formulation. We propose the world-first large-scale, open-source dataset meticulously curated for this domain called Communication System Formulation Reasoning Corpus (CSFRC). Our framework employs a two-stage training strategy: first, Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT) data to distill domain knowledge; second, a novel rule-based Reinforcement Learning (RL) algorithm, C-ReMax based on ReMax, to cultivate advanced modeling capabilities and elicit sophisticated reasoning patterns like self-correction and verification. Extensive experiments demonstrate that our model achieves state-of-the-art performance, significantly outperforming larger proprietary LLMs on diverse senerios. We will release related resources to foster further research in this area after the paper is accepted.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Human Grounded Evaluation of Large Language Models for Optical Network Automation
A human-calibrated LLM judge ranks seven LLMs for optical-network explanation generation, and Gemma3-12B tops the proposed quality-efficiency score.
Reference graph
Works this paper leans on
-
[1]
A. Hurst, A. Lerer, A. P . Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al. , “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024
arXiv 2024
-
[2]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023
arXiv 2023
-
[3]
Gemini: a family of highly capable multimodal models,
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023
arXiv 2023
-
[4]
Scaling laws for neural language models,
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020
arXiv 2001
-
[5]
Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z.-Q. Luo, “Remax: A simple, effective, and efficient reinforcement learn- ing method for aligning large language models,” arXiv preprint arXiv:2310.10505, 2023
arXiv 2023
-
[6]
Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning,
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P . Wang, X. Bi et al. , “Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[7]
HuatuoGPT, towards taming language model to be a doctor,
H. Zhang, J. Chen, F. Jiang, F. Yu, Z. Chen, G. Chen, J. Li, X. Wu, Z. Zhiyi, Q. Xiao, X. Wan, B. Wang, and H. Li, “HuatuoGPT, towards taming language model to be a doctor,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. ...
work page 2023
-
[8]
Lawllm: Intelligent legal system with legal reasoning and verifiable retrieval,
S. Yue, S. Liu, Y. Zhou, C. Shen, S. Wang, Y. Xiao, B. Li, Y. Song, X. Shen, W. Chen et al. , “Lawllm: Intelligent legal system with legal reasoning and verifiable retrieval,” in International Conference on Database Systems for Advanced Applications , 2024, pp. 304–321
work page 2024
Show all 31 references
-
[9]
Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning,
W. Chen, Q. Wang, Z. Long, X. Zhang, Z. Lu, B. Li, S. Wang, J. Xu, X. Bai, X. Huang et al. , “Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning,” CoRR, 2023
2023
-
[10]
Oceangpt: A large language model for ocean science tasks,
Z. Bi, N. Zhang, Y. Xue, Y. Ou, D. Ji, G. Zheng, and H. Chen, “Oceangpt: A large language model for ocean science tasks,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024, pp. 3357– 3372. IEEE TRANSACTION...
2024
-
[11]
Starwhisper telescope: Agent-based observation assistant system to approach ai astrophysicist,
C. Wang, X. Hu, Y. Zhang, X. Chen, P . Du, Y. Mao, R. Wang, Y. Li, Y. Wu, H. Yang et al., “Starwhisper telescope: Agent-based observation assistant system to approach ai astrophysicist,” arXiv preprint arXiv:2412.06412, 2024
2024
-
[12]
Mutaplm: Protein language modeling for mutation explanation and engi- neering,
Y. Luo, Z. Nie, M. Hong, S. Zhao, H. Zhou, and Z. Nie, “Mutaplm: Protein language modeling for mutation explanation and engi- neering,” Advances in Neural Information Processing Systems, vol. 37, pp. 79 783–79 818, 2024
2024
-
[13]
Ecomgpt: Instruction-tuning large lan- guage models with chain-of-task tasks for e-commerce,
Y. Li, S. Ma, X. Wang, S. Huang, C. Jiang, H.-T. Zheng, P . Xie, F. Huang, and Y. Jiang, “Ecomgpt: Instruction-tuning large lan- guage models with chain-of-task tasks for e-commerce,” in Pro- ceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 17, 2024, p...
2024
-
[14]
Mentallama: interpretable mental health analysis on social media with large language models,
K. Yang, T. Zhang, Z. Kuang, Q. Xie, J. Huang, and S. Ananiadou, “Mentallama: interpretable mental health analysis on social media with large language models,” in Proceedings of the ACM Web Conference 2024, 2024, pp. 4489–4500
2024
-
[15]
Doctorglm: Fine-tuning your chinese doctor is not a herculean task,
H. Xiong, S. Wang, Y. Zhu, Z. Zhao, Y. Liu, L. Huang, Q. Wang, and D. Shen, “Doctorglm: Fine-tuning your chinese doctor is not a herculean task,” arXiv preprint arXiv:2304.01097, 2023
2023 arXiv
-
[16]
Wire- lessagent: Large language model agents for intelligent wireless networks,
J. Tong, W. Guo, J. Shao, Q. Wu, Z. Li, Z. Lin, and J. Zhang, “Wire- lessagent: Large language model agents for intelligent wireless networks,” arXiv preprint arXiv:2505.01074, 2025
2025 arXiv
-
[17]
Rethinking generative semantic communication for multi-user systems with large language models,
W. Yang, Z. Xiong, S. Mao, T. Q. S. Quek, P . Zhang, M. Debbah, and R. Tafazolli, “Rethinking generative semantic communication for multi-user systems with large language models,” IEEE Wireless Communications, pp. 1–9, 2025
2025
-
[18]
Large language model enhanced multi-agent systems for 6g communications,
F. Jiang, Y. Peng, L. Dong, K. Wang, K. Yang, C. Pan, D. Niyato, and O. A. Dobre, “Large language model enhanced multi-agent systems for 6g communications,” IEEE Wireless Communications , vol. 31, no. 6, pp. 48–55, 2024
2024
-
[19]
Efficient prompting for llm-based generative internet of things,
B. Xiao, B. Kantarci, J. Kang, D. Niyato, and M. Guizani, “Efficient prompting for llm-based generative internet of things,” IEEE Internet of Things Journal, vol. 12, no. 1, pp. 778–791, 2025
2025
-
[20]
Intent-based man- agement of next-generation networks: an llm-centric approach,
A. Mekrache, A. Ksentini, and C. Verikoukis, “Intent-based man- agement of next-generation networks: an llm-centric approach,” IEEE Network, vol. 38, no. 5, pp. 29–36, 2024
2024
-
[21]
Large language model (llm)-enabled graphs in dynamic networking,
G. Sun, Y. Wang, D. Niyato, J. Wang, X. Wang, H. V . Poor, and K. B. Letaief, “Large language model (llm)-enabled graphs in dynamic networking,” IEEE Network, pp. 1–1, 2024
2024
-
[22]
Evaluating large language models for enhanced intrusion detection in internet of things networks,
E. Nwafor, U. Baskota, M. S. Parwez, J. Blackstone, and H. Olu- fowobi, “Evaluating large language models for enhanced intrusion detection in internet of things networks,” in GLOBECOM 2024 - 2024 IEEE Global Communications Conference, 2024, pp. 3358–3363
2024
-
[23]
On combining xai and llms for trustworthy zero-touch network and service management in 6g,
A. Mekrache, M. Mekki, A. Ksentini, B. Brik, and C. Verikoukis, “On combining xai and llms for trustworthy zero-touch network and service management in 6g,” IEEE Communications Magazine , vol. 63, no. 4, pp. 154–160, 2025
2025
-
[24]
Mineru: An open-source solution for precise document content extraction,
B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y. Qu, F. Shang, B. Zhang, L. Wei, Z. Sui, W. Li, B. Shi, Y. Qiao, D. Lin, and C. He, “Mineru: An open-source solution for precise document content extraction,” 2024. [Online]. Available: https://arxiv.org/abs/...
2024 arXiv
-
[25]
Opendatalab: Empowering general artificial intelligence with open datasets,
C. He, W. Li, Z. Jin, C. Xu, B. Wang, and D. Lin, “Opendatalab: Empowering general artificial intelligence with open datasets,” arXiv preprint arXiv:2407.13773, 2024
2024 arXiv
-
[26]
Binary codes capable of correcting deletions, insertions, and reversals,
V . I. Levenshtein, “Binary codes capable of correcting deletions, insertions, and reversals,” Soviet Physics Doklady, vol. 10, pp. 707– 710, 1966
1966
-
[27]
Lora: Low-rank adaptation of large language models
E. J. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al. , “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022
2022
-
[28]
Decoupled weight decay regulariza- tion,
I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in International Conference on Learning Representations (ICLR). OpenReview.net, 2019
2019
-
[29]
Qwen3 technical report,
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al. , “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025
2025 arXiv
-
[30]
Direct preference optimization: Your language model is secretly a reward model,
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, pp. 53 728–53 741, 2023
2023
-
[31]
Kto: Model alignment as prospect theoretic optimization,
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.