REVIEW 4 major objections 5 minor 299 references
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that all LLM routing policies—single-turn, multi-turn, and personalized—are instantiations of one sequential decision process made of five components, and that on the resulting xRouteBench benchmark, learned routers…
desk verdict Genuinely useful benchmark/library for LLM routing, but the headline 14.6% claim doesn't reproduce from its own table and the aggregate rankings rest on tiny, unevaluated test sets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the query–model matrix: a dense record in which every candidate in a pool of 18 models has been run on every benchmark query, scored by a task-specific metric, and priced from its token counts. This matrix is simultaneously the supervision for training routers and the test bed for evaluating them, and it is produced automatically by a three-stage pipeline (query curation, response collection, metric scoring and pricing). The organizing abstraction is the five-component scheme—context encoder $E_q$, model encoder $E_m$, scoring function $g$, decision rule $d$, and learning signal $L$—which lets single-turn, multi-turn, and personalized routers meet behind one interface; adding a router is reduced to implementing a routing method and a loss function.
What would settle it
Re-run the xRouteBench comparisons on a substantially larger held-out set (thousands of queries per track) drawn from live traffic, and replace the persona-conditioned LLM judge with human pairwise preferences collected from a large, diverse panel; if the 14.6% relative advantage of learned routers over the always-largest baseline shrinks materially, or if the top personalized router changes, the benchmark's transferability claim fails.
Extended reading notes
Core claim
The paper's central claim is that every LLM router is an instance of a sequential decision process: at each step the router observes a state $s_t = (q, u, h_t)$ consisting of the query, optional user context, and interaction history, and either dispatches the state to a candidate model $m \in \mathcal{M}$ or terminates and aggregates the collected responses. The router's goal is to maximize $\mathbb{E}[\mathrm{perf}(y|q) - \lambda \cdot c(\tau)]$, quality minus a cost penalty. The paper claims that on xRouteBench—4,767 queries spanning generic, memory, vision, time-series, and personalized tasks—learned routers beat the strongest fixed-model baseline by 14.6% relatively, that router rankings reverse in favor of lightweight designs as cost weight rises, and that user-conditioned routing improves personalization consistently, though the best personalized design differs between an LLM judge and real users.
Load-bearing premise
The empirical case assumes that xRouteBench's 4,767 queries, several tracks having only 13–127 test instances, and the fixed 18-model price list yield router rankings that transfer to real deployments, and that a persona-conditioned LLM judge validated on just 15 users and 234 preference records is faithful to human preference.
Editorial extensions
If this is right
- Adopting the five-component formulation means a new router is added by writing a routing method and a loss function; data construction, training, inference, and evaluation then apply unchanged.
- Learned routers can beat the always-largest baseline by 14.6% relatively because many queries the largest model answers incorrectly are solved by smaller, cheaper models.
- Router rankings reverse as the cost weight increases, so the right router for a deployment depends on its operating point; lightweight routers become more competitive under tight cost constraints.
- Multi-turn routing does not consistently beat single-turn routing; gains from decomposition and aggregation depend on the capability of the base model that performs them.
- User-conditioned routing improves personalization, but how user context is encoded matters: the top router under an LLM persona judge (GMTRouter) was not the top router on real-user Slack feedback (PersonalizedRouter).
Reading between the lines
- (Editorial) The same five-component formulation could transfer to non-LLM model-selection problems—routing between multimodal encoders, code interpreters, or database agents—because only the context and model encoders plus scoring function need re-specification.
- (Editorial) Several xRouteBench tracks are very small (13–127 test instances), so the reported margins on those tracks should be treated as preliminary until the test sets are enlarged; rankings there could flip with additional data.
- (Editorial) The success of a persona-conditioned LLM judge at reproducing personalized rankings suggests a scalable substitute for human preference collection; a testable extension is to audit judge fidelity continuously against a small always-on human panel.
- (Editorial) The finding that multi-turn routing adds cost without consistent gain points to adaptive early stopping as the next lever: a router that decides when to stop, rather than only which model to call, could recover the lost efficiency.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified formulation of LLM routing as a sequential decision process with five components (context encoder, model encoder, scoring function, decision rule, learning signal), organizes existing routers into single-turn, multi-turn, and personalized families, and builds an open-source infrastructure (LLMRouter) with more than 16 implemented routers and an automated data-construction pipeline. It introduces xRouteBench, a five-track benchmark with 4,767 test queries across generic text, memory-augmented, vision, time-series, and personalized settings. The empirical study claims that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, that router rankings reverse under cost constraints, that multi-turn routing does not consistently outperform single-turn routing, and that user-conditioned routing improves personalization. The paper also reports a small real-user Slack deployment study and a multi-agent system evaluation.
Significance. If the empirical claims hold, the paper would provide a useful common ground for LLM routing research: a principled component-wise formulation, a shared evaluation protocol, a multi-scenario benchmark, and a library with a uniform interface, plus deployment evidence from a live messaging platform. The inclusion of the real-user Slack study and the multi-agent evaluation is a genuine strength, as is the explicit reporting of limitations such as the sensitivity of multi-turn routers to their base model. The infrastructure contribution is valuable regardless of the exact headline numbers: the formulation and modular library should lower the cost of fair comparison. However, the central quantitative claims currently rest on aggregate numbers whose derivation is not stated and on comparisons without uncertainty quantification, so the empirical conclusions, as presented, are not yet fully supported.
major comments (4)
- [Abstract; §5.2, Table 2] The headline claim that learned routers outperform the strongest fixed-model baseline by 14.6% relatively is not reproducible from Table 2 as presented. The Table 2 "Avg" column is the unweighted arithmetic mean of the seven reported test sets; for GraphRouter versus Largest-LLM this gives (45.46/38.72 − 1) ≈ 17.4%, not 14.6%. A query-weighted average over all 4,767 xRouteBench queries gives approximately 14.6%, but no weighted aggregation rule is stated anywhere in the paper, and Table 2's Avg is the only aggregate shown. Since this number is the paper's central empirical claim, the manuscript must explicitly define the aggregation used for the headline and report both unweighted and query-weighted results.
- [§5.2, Tables 2–5, Table 6] No confidence intervals, bootstrap intervals, or significance tests are reported for any of the central comparisons, despite several very small test sets: Charades-Ego has 27 test queries, Geometry3K 61, MathVista 100, LongMemEval 101, and TimeSeries 127 (Table 6). The GraphRouter-versus-Largest-LLM advantages on MathVista and TimeSeries are roughly 17 points on 100 and 127 queries, about 2.4 to 2.5 standard errors for a difference of proportions; the aggregate rankings (GraphRouter 45.46 vs SVMRouter 45.10 in Table 2, and GMTRouter 68.78 vs EloRouter 66.40 in Table 3) are within plausible sampling noise. Without uncertainty quantification, the claims of "best average" performance, of the cost-sweep ranking reversals in Figure 5, and of "consistent personalization gains" are not statistically supported as reported.
- [§3 Personalized track; §5.4, Tables 3–4, Appendix D] The personalized track's supervision and its evaluation both come from the same persona-conditioned DeepSeek-V3.1 judge, so the reported gains of GMTRouter and PersonalizedRouter over user-agnostic routers could reflect overfitting to that judge rather than to human preferences. The paper wisely includes a real-user Slack study as independent evidence, but that study contains only 234 pairwise records from 15 users (Appendix D), and its ranking differs from the persona-judge ranking (GMTRouter drops from first under the judge to sixth under real users). The manuscript should report judge–human agreement statistics or an explicit calibration, and should present the user study with per-user variance, before claiming that user-conditioned routing "consistently improves personalization."
- [§5.2, Table 2] The finding that "multi-turn routing does not consistently outperform single-turn routing" is confounded by the choice of base model: all multi-turn routers use Qwen2.5-3B-Instruct for decomposition and aggregation, while single-turn routers do not carry that dependency. The very low multi-turn scores (roughly 22–23 in Table 2) may reflect the capability of the 3B base model as much as the routing formulation. An ablation with a stronger base model, or an explicit justification for why the 3B model is the correct comparison point, is needed before this can be stated as a general conclusion about the multi-turn family.
minor comments (5)
- [Table 2] The "Avg" column should be explicitly defined as the arithmetic mean of the seven track scores; the current label is ambiguous and directly relevant to the reproducibility issue above.
- [§5.3, Figure 5] The rank matrices in Figure 5 are difficult to read at printed size and do not show the underlying reward or cost values; a table of the weighted rewards per β setting would make the cost-sweep claims easier to verify.
- [Appendix C] Appendix C mentions a "17-model no-cogito pool" that drops the most expensive model, but the main experiments do not say whether this pool is used anywhere; this should be clarified or removed.
- [§5.1 and Table 8] The router is referred to as both "CausalLM" and "CausalLM Router" in different places; please use one consistent name throughout.
- [§2.1, Eq. (1)] The relationship between the abstract performance–cost objective in Eq. (1) and the experimental weighted reward α·perf − β·cost in §5.1 is not spelled out; a short sentence connecting λ to (α, β) would prevent confusion.
Circularity Check
Minor self-definitional component in the cost-trade-off ranking; central benchmark and personalization claims remain externally grounded.
-
self definitional
[§5.1 Evaluation Protocol and §5.3 Performance–Cost Trade-offs (Eq. 1)]
"We score each router by a weighted reward α·perf−β·cost. We sweep five weight settings from the quality-only (α,β)=(1.0,0.0) to the heavily cost-weighted (0.2,0.8). ... Figure 5 ranks the routers by reward within each category as the cost weight β grows, and the rankings shift dramatically along the sweep."
The rankings in Figure 5 are computed from the same weighted reward α·perf−β·cost whose cost weight β is swept. Thus the qualitative finding that 'router rankings reverse in favor of lightweight designs under tighter cost constraints' is entailed by the definition of the ranking metric: raising β mechanically increases the cost penalty in the scored reward. The observation restates the evaluation objective rather than being an independent empirical discovery; only which specific routers occupy which ranks is data-dependent. This is a secondary finding, not the central derivation, and does not affect the main benchmark comparisons.
full rationale
The paper's central contributions are a formulation, a benchmark, and a library. The unified formulation is a taxonomy that organizes existing routers by five components; it makes no predictive claim that could reduce to its own definition. xRouteBench is constructed from external source benchmarks with standard task metrics, and routers are trained on training splits and evaluated on held-out test splits, which is standard supervised evaluation rather than a circular fit. The personalized track does use a persona-conditioned DeepSeek-V3.1 judge for both supervision and evaluation, which would be a closed loop, but the paper independently validates personalization against 15 real users and 234 Slack preference records and explicitly reports that the simulated ranking does not fully transfer, so the personalization claim has external grounding. Self-citations (GraphRouter, PersonalizedRouter, GMTRouter, Router-R1, GraphPlanner) appear as implemented baselines whose performance is measured in-paper, not as unverified premises supporting the conclusions. The only definitional component is the cost-trade-off ranking described in the step above, which is a secondary finding. The unexplained 14.6% relative improvement (which matches the Generic-track RouterDC gain rather than Table 2's average) is a reporting and aggregation inconsistency, a correctness concern rather than circularity. Overall, no load-bearing derivation reduces to its inputs.
Assumptions & free parameters
free parameters (3)
- Objective weights (alpha, beta) =
Swept from (1.0, 0.0) to (0.2, 0.8)
- Memory retrieval top-k =
5
- Persona count for personalized track =
200
assumptions (4)
- domain assumption The linear reward in Eq. 1 (perf minus lambda times cost, or alpha times perf minus beta times cost) is a valid deployment utility.
- domain assumption Persona-conditioned DeepSeek-V3.1 judge labels stand in for real user preferences on the personalized track.
- domain assumption Turning images, video frames, and time series into text with fixed captioners preserves the routing-relevant information.
- domain assumption The fixed 18-model candidate pool and listed prices are representative of real deployment conditions.
Cite this review
Pith. "Pith review of LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers." pith.science (2026). https://pith.science/paper/SZ4TJVTV
@misc{pith2026260806867,
author = {Pith},
title = {Pith review of: LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZ4TJVTV}},
note = {Machine review of arXiv:2608.06867}
}
read the original abstract
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2410.14752 , year=
TimeSeriesExam: A time series understanding exam , author=. arXiv preprint arXiv:2410.14752 , year=
-
[2]
arXiv preprint arXiv:2405.14755 , year=
Large language models can be zero-shot anomaly detectors for time series? , author=. arXiv preprint arXiv:2405.14755 , year=
-
[3]
International Conference on Machine Learning , year=
MOMENT: A Family of Open Time-series Foundation Models , author=. International Conference on Machine Learning , year=
-
[4]
Mixllm: Dynamic routing in mixed large language models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2025
-
[5]
arXiv preprint arXiv:2509.04439 , year=
ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory , author=. arXiv preprint arXiv:2509.04439 , year=
-
[6]
arXiv preprint arXiv:2506.17587 , year=
Hallurnn: Mitigating hallucinations via recurrent cross-layer reasoning in large vision-language models , author=. arXiv preprint arXiv:2506.17587 , year=
-
[7]
arXiv preprint arXiv:2403.17297 , year=
Internlm2 technical report , author=. arXiv preprint arXiv:2403.17297 , year=
-
[8]
arXiv preprint arXiv:2502.00816 , year=
Sundial: A family of highly capable time series foundation models , author=. arXiv preprint arXiv:2502.00816 , year=
Show all 299 references
-
[9]
arXiv preprint arXiv:2502.21245 , year=
Timesbert: A bert-style foundation model for time series understanding , author=. arXiv preprint arXiv:2502.21245 , year=
-
[10]
arXiv preprint arXiv:2303.08774 , year=
Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=
-
[11]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Gpt4mts: Prompt-based large language model for multimodal time-series forecasting , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[12]
Advances in Neural Information Processing Systems , volume=
Sharegpt4video: Improving video understanding and generation with better captions , author=. Advances in Neural Information Processing Systems , volume=
-
[13]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Conzic: Controllable zero-shot image captioning by sampling-based polishing , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[14]
Proceedings of the 31st ACM International Conference on Multimedia , pages=
Beyond generic: Enhancing image captioning with real-world knowledge using vision-language pre-training model , author=. Proceedings of the 31st ACM International Conference on Multimedia , pages=
-
[15]
arXiv preprint arXiv:2605.17672 , year=
Stop When Reasoning Converges: Semantic-Preserving Early Exit for Reasoning Models , author=. arXiv preprint arXiv:2605.17672 , year=
-
[16]
arXiv preprint arXiv:2503.12329 , year=
Caparena: Benchmarking and analyzing detailed image captioning in the llm era , author=. arXiv preprint arXiv:2503.12329 , year=
-
[17]
arXiv preprint arXiv:2604.13016 , year=
Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe , author=. arXiv preprint arXiv:2604.13016 , year=
-
[18]
2025 , howpublished =
LLMRouter: An Open-Source Library for LLM Routing , author =. 2025 , howpublished =
2025
-
[19]
arXiv preprint arXiv:2604.23626 , year=
GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMs , author=. arXiv preprint arXiv:2604.23626 , year=
-
[20]
arXiv preprint arXiv:2604.03128 , year=
Self-distilled rlvr , author=. arXiv preprint arXiv:2604.03128 , year=
-
[21]
arXiv preprint arXiv:2602.17149 , year=
Timeomni-vl: Unified models for time series understanding and generation , author=. arXiv preprint arXiv:2602.17149 , year=
-
[22]
arXiv preprint arXiv:2601.19897 , year=
Self-Distillation Enables Continual Learning , author=. arXiv preprint arXiv:2601.19897 , year=
-
[23]
arXiv preprint arXiv:2406.05673 , year=
Flow of reasoning: Training llms for divergent reasoning with minimal examples , author=. arXiv preprint arXiv:2406.05673 , year=
-
[24]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[25]
arXiv preprint arXiv:2601.18734 , year=
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=
-
[26]
Advances in Neural Information Processing Systems , volume=
Revisiting reinforcement learning for llm reasoning from a cross-domain perspective , author=. Advances in Neural Information Processing Systems , volume=
-
[27]
Advances in neural information processing systems , volume=
Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=
-
[28]
arXiv preprint arXiv:2404.16563 , year=
Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark , author=. arXiv preprint arXiv:2404.16563 , year=
-
[29]
arXiv preprint arXiv:2305.14992 , year=
Reasoning with language model is planning with world model , author=. arXiv preprint arXiv:2305.14992 , year=
-
[30]
arXiv preprint arXiv:2402.16132 , year=
Lstprompt: Large language models as zero-shot time series forecasters by long-short-term prompting , author=. arXiv preprint arXiv:2402.16132 , year=
-
[31]
Advances in Neural Information Processing Systems , volume=
Large language models are zero-shot time series forecasters , author=. Advances in Neural Information Processing Systems , volume=
-
[32]
Deep Generative Models for Health Workshop NeurIPS 2023 , year=
Jolt: Jointly learned representations of language and time-series , author=. Deep Generative Models for Health Workshop NeurIPS 2023 , year=
2023
-
[33]
arXiv preprint arXiv:2312.01454 , year=
D-bot: Database diagnosis system using large language models , author=. arXiv preprint arXiv:2312.01454 , year=
-
[34]
Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages=
Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models , author=. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages=
-
[35]
arXiv preprint arXiv:2208.03938 , year=
Constructing large-scale real-world benchmark datasets for aiops , author=. arXiv preprint arXiv:2208.03938 , year=
-
[36]
Neurocomputing , volume=
Unsupervised real-time anomaly detection for streaming data , author=. Neurocomputing , volume=. 2017 , publisher=
2017
-
[37]
Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages=
Actionable and interpretable fault localization for recurring failures in online service systems , author=. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages=
-
[38]
arXiv preprint arXiv:2404.11757 , year=
Language Models Still Struggle to Zero-shot Reason about Time Series , author=. arXiv preprint arXiv:2404.11757 , year=
-
[39]
Proceedings of the International Conference on Internet-of-Things Design and Implementation , pages=
Deepsqa: Understanding sensor data via question answering , author=. Proceedings of the International Conference on Internet-of-Things Design and Implementation , pages=
-
[40]
5-vl technical report , author=
Qwen2. 5-vl technical report , author=. arXiv preprint arXiv:2502.13923 , year=
-
[41]
Applied soft computing , volume=
Financial time series forecasting with deep learning: A systematic literature review: 2005--2019 , author=. Applied soft computing , volume=. 2020 , publisher=
2005
-
[42]
arXiv preprint arXiv:2401.13601 , year=
Mm-llms: Recent advances in multimodal large language models , author=. arXiv preprint arXiv:2401.13601 , year=
-
[43]
National Science Review , pages=
A survey on multimodal large language models , author=. National Science Review , pages=. 2024 , publisher=
2024
-
[44]
Advances in neural information processing systems , volume=
Visual instruction tuning , author=. Advances in neural information processing systems , volume=
-
[45]
arXiv preprint arXiv:2308.12966 , year=
Qwen-vl: A frontier large vision-language model with versatile abilities , author=. arXiv preprint arXiv:2308.12966 , year=
-
[46]
2015 , publisher=
Time series analysis: forecasting and control , author=. 2015 , publisher=
2015
-
[47]
International conference on machine learning , pages=
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=
2023
-
[48]
arXiv preprint arXiv:2501.07335 , year=
TempoGPT: Enhancing Temporal Reasoning via Quantizing Embedding , author=. arXiv preprint arXiv:2501.07335 , year=
-
[49]
arXiv preprint arXiv:2306.05424 , year=
Video-chatgpt: Towards detailed video understanding via large vision and language models , author=. arXiv preprint arXiv:2306.05424 , year=
-
[50]
arXiv preprint arXiv:2306.02858 , year=
Video-llama: An instruction-tuned audio-visual language model for video understanding , author=. arXiv preprint arXiv:2306.02858 , year=
-
[51]
arXiv preprint arXiv:2305.06355 , year=
Videochat: Chat-centric video understanding , author=. arXiv preprint arXiv:2305.06355 , year=
-
[52]
arXiv preprint arXiv:2311.07919 , year=
Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models , author=. arXiv preprint arXiv:2311.07919 , year=
-
[53]
arXiv preprint arXiv:2306.12925 , year=
Audiopalm: A large language model that can speak and listen , author=. arXiv preprint arXiv:2306.12925 , year=
-
[54]
Proceedings of the ACM on Web Conference 2024 , pages=
GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks , author=. Proceedings of the ACM on Web Conference 2024 , pages=
2024
-
[55]
IEEE Transactions on Knowledge and Data Engineering , year=
Unifying large language models and knowledge graphs: A roadmap , author=. IEEE Transactions on Knowledge and Data Engineering , year=
-
[56]
arXiv preprint arXiv:2409.05840 , year=
Mmevol: Empowering multimodal large language models with evol-instruct , author=. arXiv preprint arXiv:2409.05840 , year=
-
[57]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[58]
arXiv preprint arXiv:2409.11376 , year=
Towards time series reasoning with llms , author=. arXiv preprint arXiv:2409.11376 , year=
-
[59]
arXiv preprint arXiv:2402.02713 , year=
Position paper: What can large language models tell us about time series analysis , author=. arXiv preprint arXiv:2402.02713 , year=
-
[60]
Forty-first International Conference on Machine Learning , year=
A decoder-only foundation model for time-series forecasting , author=. Forty-first International Conference on Machine Learning , year=
-
[61]
arXiv preprint arXiv:2412.03104 , year=
ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning , author=. arXiv preprint arXiv:2412.03104 , year=
-
[62]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Chattime: A unified multimodal time series foundation model bridging numerical and textual data , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[63]
5: Visual Agentic Intelligence , author=
Kimi K2. 5: Visual Agentic Intelligence , author=. arXiv preprint arXiv:2602.02276 , year=
-
[64]
arXiv preprint arXiv:2506.05176 , year=
Qwen3 embedding: Advancing text embedding and reranking through foundation models , author=. arXiv preprint arXiv:2506.05176 , year=
-
[66]
arXiv preprint arXiv:2508.10925 , year=
gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year=
-
[67]
arXiv preprint arXiv:2505.09388 , year=
Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=
-
[68]
2025 , note =
Meta Platforms , title =. 2025 , note =
2025
-
[69]
Proceedings of the web conference 2020 , pages=
Heterogeneous graph transformer , author=. Proceedings of the web conference 2020 , pages=
2020
-
[70]
arXiv preprint arXiv:2305.05176 , year=
Frugalgpt: How to use large language models while reducing cost and improving performance , author=. arXiv preprint arXiv:2305.05176 , year=
-
[71]
arXiv preprint arXiv:2511.21631 , year=
Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=
-
[72]
arXiv preprint arXiv:2507.01006 , year=
Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning , author=. arXiv preprint arXiv:2507.01006 , year=
-
[73]
arXiv preprint arXiv:1609.02907 , year=
Semi-supervised classification with graph convolutional networks , author=. arXiv preprint arXiv:1609.02907 , year=
-
[74]
arXiv preprint arXiv:2601.19151 , year=
TS-Debate: Multimodal Collaborative Debate for Zero-Shot Time Series Reasoning , author=. arXiv preprint arXiv:2601.19151 , year=
-
[75]
arXiv preprint arXiv:2309.15789 , year=
Large language model routing with benchmark datasets , author=. arXiv preprint arXiv:2309.15789 , year=
-
[76]
arXiv preprint arXiv:2503.01013 , year=
Timexl: Explainable multi-modal time series prediction with llm-in-the-loop , author=. arXiv preprint arXiv:2503.01013 , year=
-
[77]
arXiv preprint arXiv:2502.01477 , year=
Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluation , author=. arXiv preprint arXiv:2502.01477 , year=
-
[78]
arXiv preprint arXiv:1412.6980 , year=
Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=
-
[79]
North American Chapter of the Association for Computational Linguistics , year=
A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization , author=. North American Chapter of the Association for Computational Linguistics , year=
-
[80]
arXiv preprint arXiv:2405.16587 , year=
Cost-effective online multi-llm selection with versatile reward models , author=. arXiv preprint arXiv:2405.16587 , year=
-
[81]
arXiv preprint arXiv:1903.02428 , year=
Fast graph representation learning with PyTorch Geometric , author=. arXiv preprint arXiv:1903.02428 , year=
1903 arXiv
-
[82]
IEEE Signal Processing Letters , volume=
Multivariate time series imputation with transformers , author=. IEEE Signal Processing Letters , volume=. 2022 , publisher=
2022
-
[83]
Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia , pages=
Conversational interfaces in iot ecosystems: where we are, what is still missing , author=. Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia , pages=
-
[84]
Machine Learning for Health (ML4H) , pages=
Zero-shot ECG diagnosis with large language models and retrieval-augmented generation , author=. Machine Learning for Health (ML4H) , pages=. 2023 , organization=
2023
-
[85]
Advances in Neural Information Processing Systems , volume=
Ecg-qa: A comprehensive question answering dataset combined with electrocardiogram , author=. Advances in Neural Information Processing Systems , volume=
-
[86]
Joint conference of the information retrieval communities in europe (circle 2020) , volume=
Event detection and time series alignment to improve stock market forecasting , author=. Joint conference of the information retrieval communities in europe (circle 2020) , volume=. 2020 , organization=
2020
-
[87]
arXiv preprint arXiv:2407.03689 , year=
Text2timeseries: Enhancing financial forecasting through time series prediction updates with event-driven insights from large language models , author=. arXiv preprint arXiv:2407.03689 , year=
-
[88]
International Journal of Machine Learning and Cybernetics , pages=
Open-ti: Open traffic intelligence with augmented language model , author=. International Journal of Machine Learning and Cybernetics , pages=. 2024 , publisher=
2024
-
[89]
arXiv preprint arXiv:2312.16044 , year=
Large language models as traffic signal control agents: Capacity and opportunity , author=. arXiv preprint arXiv:2312.16044 , year=
-
[90]
Advances in Neural Information Processing Systems , volume=
Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in Neural Information Processing Systems , volume=
-
[91]
arXiv preprint arXiv:2210.03629 , year=
React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year=
-
[92]
arXiv preprint arXiv:2310.01728 , year=
Time-llm: Time series forecasting by reprogramming large language models , author=. arXiv preprint arXiv:2310.01728 , year=
-
[93]
arXiv preprint arXiv:2411.02465 , year=
See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzers , author=. arXiv preprint arXiv:2411.02465 , year=
-
[94]
arXiv preprint arXiv:2402.10350 , year=
Large language models for forecasting and anomaly detection: A systematic literature review , author=. arXiv preprint arXiv:2402.10350 , year=
-
[95]
arXiv preprint arXiv:2308.08469 , year=
Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms , author=. arXiv preprint arXiv:2308.08469 , year=
-
[96]
arXiv preprint arXiv:2406.08627 , year=
Time-MMD: A New Multi-Domain Multimodal Dataset for Time Series Analysis , author=. arXiv preprint arXiv:2406.08627 , year=
-
[97]
arXiv preprint arXiv:2503.16858 , year=
MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering , author=. arXiv preprint arXiv:2503.16858 , year=
-
[98]
arXiv preprint arXiv:2405.15370 , year=
Large language models can deliver accurate and interpretable time series anomaly detection , author=. arXiv preprint arXiv:2405.15370 , year=
-
[99]
arXiv preprint arXiv:2408.17253 , year=
VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters , author=. arXiv preprint arXiv:2408.17253 , year=
-
[100]
arXiv preprint arXiv:2310.04948 , year=
Tempo: Prompt-based generative pre-trained transformer for time series forecasting , author=. arXiv preprint arXiv:2310.04948 , year=
-
[101]
Advances in neural information processing systems , volume=
One fits all: Power general time series analysis by pretrained lm , author=. Advances in neural information processing systems , volume=
-
[102]
IEEE Transactions on Intelligent Transportation Systems , volume=
Trend modeling for traffic time series analysis: An integrated study , author=. IEEE Transactions on Intelligent Transportation Systems , volume=. 2015 , publisher=
2015
-
[103]
Philosophical Transactions of the Royal Society A , volume=
Time-series forecasting with deep learning: a survey , author=. Philosophical Transactions of the Royal Society A , volume=. 2021 , publisher=
2021
-
[104]
Academic pediatrics , volume=
Use of interrupted time series analysis in evaluating health care quality improvements , author=. Academic pediatrics , volume=. 2013 , publisher=
2013
-
[105]
Energy policy , volume=
Electricity consumption and economic growth: a time series experience for 17 African countries , author=. Energy policy , volume=. 2006 , publisher=
2006
-
[106]
arXiv preprint arXiv:2409.17515 , year=
From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection , author=. arXiv preprint arXiv:2409.17515 , year=
-
[107]
arXiv preprint arXiv:2402.00607 , year=
Are Synthetic Time-series Data Really not as Good as Real Data? , author=. arXiv preprint arXiv:2402.00607 , year=
-
[108]
2018 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm) , pages=
Generative adversarial network for synthetic time series data generation in smart grids , author=. 2018 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm) , pages=. 2018 , organization=
2018
-
[109]
arXiv preprint arXiv:2308.00393 , year=
A Survey of Time Series Anomaly Detection Methods in the AIOps Domain , author=. arXiv preprint arXiv:2308.00393 , year=
-
[110]
OpenAI GPT-4o. , url =
-
[111]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=
2024
-
[112]
J Off Stat , volume=
STL: A seasonal-trend decomposition procedure based on loess , author=. J Off Stat , volume=
-
[113]
arXiv preprint arXiv:2304.12244 , year=
Wizardlm: Empowering large language models to follow complex instructions , author=. arXiv preprint arXiv:2304.12244 , year=
-
[114]
Data Mining and Knowledge Discovery , volume=
ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels , author=. Data Mining and Knowledge Discovery , volume=. 2020 , publisher=
2020
-
[115]
arXiv preprint arXiv:2309.15217 , year=
Ragas: Automated evaluation of retrieval augmented generation , author=. arXiv preprint arXiv:2309.15217 , year=
-
[116]
, author=
Synthetic data could be better than real data. , author=. Nature , year=
-
[117]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Time series contrastive learning with information-aware augmentations , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[118]
arXiv preprint arXiv:2407.06426 , year=
DebUnc: mitigating hallucinations in large language model agent communication with uncertainty estimations , author=. arXiv preprint arXiv:2407.06426 , year=
-
[119]
Xiao He and Ye Li and Jian Tan and Bin Wu and Feifei Li , title =. Proc. 2023 , url =. doi:10.14778/3583140.3583155 , timestamp =
2023
-
[120]
, year =
Weather Dataset. , year =
-
[121]
, year =
MCQ2 Dataset. , year =
-
[122]
, year =
Anomaly Detection Toolkit. , year =
-
[123]
European Conference on Computer Vision , pages=
Veclip: Improving clip training via visual-enriched captions , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[124]
Advances in Neural Information Processing Systems , volume=
Improving clip training with language rewrites , author=. Advances in Neural Information Processing Systems , volume=
-
[125]
arXiv preprint arXiv:2410.02740 , year=
Revisit large-scale image-caption data in pre-training multimodal foundation models , author=. arXiv preprint arXiv:2410.02740 , year=
-
[126]
arXiv preprint arXiv:2405.13326 , year=
Mosaic-IT: Free Compositional Data Augmentation Improves Instruction Tuning , author=. arXiv preprint arXiv:2405.13326 , year=
-
[127]
, year =
Qwen2.5-14B-Instruct Model. , year =
-
[128]
Advances in neural information processing systems , volume=
Attention is all you need , author=. Advances in neural information processing systems , volume=
-
[129]
International conference on machine learning , pages=
Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=
2021
-
[130]
International conference on machine learning , pages=
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=
2022
-
[131]
Advances in neural information processing systems , volume=
Adversarial sparse transformer for time series forecasting , author=. Advances in neural information processing systems , volume=
-
[132]
Advances in Neural Information Processing Systems , volume=
Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting , author=. Advances in Neural Information Processing Systems , volume=
-
[133]
arXiv preprint arXiv:2201.12740 , year=
FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting , author=. arXiv preprint arXiv:2201.12740 , year=
-
[134]
International Journal of Forecasting , volume=
Temporal fusion transformers for interpretable multi-horizon time series forecasting , author=. International Journal of Forecasting , volume=. 2021 , publisher=
2021
-
[135]
International Conference on Learning Representations , year=
Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting , author=. International Conference on Learning Representations , year=
-
[136]
arXiv preprint arXiv:2205.13504 , year=
Are Transformers Effective for Time Series Forecasting? , author=. arXiv preprint arXiv:2205.13504 , year=
-
[137]
The Twelfth International Conference on Learning Representations , year=
Transformer-modulated diffusion models for probabilistic multivariate time series forecasting , author=. The Twelfth International Conference on Learning Representations , year=
-
[138]
International Conference on Machine Learning , pages=
Prototype-oriented unsupervised anomaly detection for multivariate time series , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[139]
arXiv preprint arXiv:2406.14045 , year=
Understanding Different Design Choices in Training Large Time Series Models , author=. arXiv preprint arXiv:2406.14045 , year=
-
[140]
arXiv preprint arXiv:2308.08241 , year=
Test: Text prototype aligned embedding to activate llm's ability for time series , author=. arXiv preprint arXiv:2308.08241 , year=
-
[141]
Pan, Zijie and Jiang, Yushan and Garg, Sahil and Schneider, Anderson and Nevmyvaka, Yuriy and Song, Dongjin , booktitle =. S^2. 2024 , editor =
2024
-
[142]
Biomedical Signal Processing and Control , volume=
Time series signal forecasting using artificial neural networks: An application on ECG signal , author=. Biomedical Signal Processing and Control , volume=. 2022 , publisher=
2022
-
[143]
5 technical report , author=
Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=
-
[144]
arXiv preprint arXiv:2410.10469 , year=
Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts , author=. arXiv preprint arXiv:2410.10469 , year=
-
[145]
arXiv preprint arXiv:2406.11903 , year=
A survey of large language models for financial applications: Progress, prospects and challenges , author=. arXiv preprint arXiv:2406.11903 , year=
-
[146]
Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages=
Density-aware temporal attentive step-wise diffusion model for medical time series imputation , author=. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages=
-
[147]
arXiv preprint arXiv:2501.18592 , year=
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models , author=. arXiv preprint arXiv:2501.18592 , year=
-
[148]
arXiv preprint arXiv:2407.19795 , year=
VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks , author=. arXiv preprint arXiv:2407.19795 , year=
-
[149]
arXiv preprint arXiv:2502.04395 , year=
Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting , author=. arXiv preprint arXiv:2502.04395 , year=
-
[150]
Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages=
A transformer-based framework for multivariate time series representation learning , author=. Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages=
-
[151]
arXiv preprint arXiv:2211.14730 , year=
A time series is worth 64 words: Long-term forecasting with transformers , author=. arXiv preprint arXiv:2211.14730 , year=
-
[152]
arXiv preprint arXiv:2402.02368 , year=
Timer: Generative pre-trained transformers are large time series models , author=. arXiv preprint arXiv:2402.02368 , year=
-
[153]
Advances in Neural Information Processing Systems , volume=
Tiny time mixers (ttms): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series , author=. Advances in Neural Information Processing Systems , volume=
-
[154]
arXiv preprint arXiv:2407.21783 , year=
The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=
-
[155]
The Claude 3 Model Family: Opus, Sonnet, Haiku , author=
-
[156]
arXiv preprint arXiv:2412.19437 , year=
Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=
-
[157]
European Conference on Computer Vision , pages=
Sharegpt4v: Improving large multi-modal models with better captions , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[158]
arXiv preprint arXiv:2408.01800 , year=
MiniCPM-V: A GPT-4V Level MLLM on Your Phone , author=. arXiv preprint arXiv:2408.01800 , year=
-
[159]
arXiv preprint arXiv:2504.10479 , year=
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models , author=. arXiv preprint arXiv:2504.10479 , year=
-
[160]
LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild , url=
Li, Bo and Zhang, Kaichen and Zhang, Hao and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Yuanhan and Liu, Ziwei and Li, Chunyuan , month=. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild , url=
-
[161]
Language Learning , volume=
Measuring lexical diversity in texts: The twofold length problem , author=. Language Learning , volume=. 2024 , publisher=
2024
-
[162]
Proceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation , pages=
Are Time Series Foundation Models Ready to Revolutionize Predictive Building Analytics? , author=. Proceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation , pages=
-
[163]
arXiv preprint arXiv:2501.03747 , year=
Context-alignment: Activating and enhancing llm capabilities in time series , author=. arXiv preprint arXiv:2501.03747 , year=
-
[164]
arXiv preprint arXiv:2505.10083 , year=
ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Data , author=. arXiv preprint arXiv:2505.10083 , year=
-
[165]
The 41st international ACM SIGIR conference on research & development in information retrieval , pages=
Texygen: A benchmarking platform for text generation models , author=. The 41st international ACM SIGIR conference on research & development in information retrieval , pages=
-
[166]
arXiv preprint arXiv:2503.01743 , year=
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras , author=. arXiv preprint arXiv:2503.01743 , year=
-
[167]
arXiv preprint arXiv:2401.04088 , year=
Mixtral of experts , author=. arXiv preprint arXiv:2401.04088 , year=
-
[168]
arXiv preprint arXiv:2406.12793 , year=
Chatglm: A family of large language models from glm-130b to glm-4 all tools , author=. arXiv preprint arXiv:2406.12793 , year=
-
[169]
arXiv preprint arXiv:2409.16040 , year=
Time-moe: Billion-scale time series foundation models with mixture of experts , author=. arXiv preprint arXiv:2409.16040 , year=
-
[170]
arXiv preprint arXiv:2403.07815 , year=
Chronos: Learning the language of time series , author=. arXiv preprint arXiv:2403.07815 , year=
-
[171]
2024 , publisher=
Unified training of universal time series forecasting transformers , author=. 2024 , publisher=
2024
-
[172]
Proceedings of the ACM/SPEC International Conference on Performance Engineering , pages=
Libra: A benchmark for time series forecasting methods , author=. Proceedings of the ACM/SPEC International Conference on Performance Engineering , pages=
-
[173]
arXiv preprint arXiv:2403.20150 , year=
Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods , author=. arXiv preprint arXiv:2403.20150 , year=
-
[174]
Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V
Tsfm-bench: A comprehensive and unified benchmark of foundation models for time series forecasting , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=
-
[175]
arXiv preprint arXiv:2502.18834 , year=
Fintsb: A comprehensive and practical benchmark for financial time series forecasting , author=. arXiv preprint arXiv:2502.18834 , year=
-
[176]
arXiv preprint arXiv:2407.13278 , year=
Deep time series models: A comprehensive survey and benchmark , author=. arXiv preprint arXiv:2407.13278 , year=
-
[177]
arXiv preprint arXiv:2105.06643 , year=
Monash time series forecasting archive , author=. arXiv preprint arXiv:2105.06643 , year=
-
[178]
Data mining and knowledge discovery , volume=
Deep learning for time series classification: a review , author=. Data mining and knowledge discovery , volume=. 2019 , publisher=
2019
-
[179]
arXiv preprint arXiv:2007.13156 , year=
Benchmarking multivariate time series classification algorithms , author=. arXiv preprint arXiv:2007.13156 , year=
2007 arXiv
-
[180]
arXiv preprint arXiv:2410.05440 , year=
Can LLMs understand time series anomalies? , author=. arXiv preprint arXiv:2410.05440 , year=
-
[181]
Proceedings of the VLDB Endowment , volume=
Timeeval: A benchmarking toolkit for time series anomaly detection algorithms , author=. Proceedings of the VLDB Endowment , volume=. 2022 , publisher=
2022
-
[182]
Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1) , year=
Revisiting time series outlier detection: Definitions and benchmarks , author=. Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1) , year=
-
[183]
arXiv preprint arXiv:2406.12747 , year=
Tsi-bench: Benchmarking time series imputation , author=. arXiv preprint arXiv:2406.12747 , year=
-
[184]
Journal of biomedical informatics , volume=
Deep imputation of missing values in time series health data: A review with benchmarking , author=. Journal of biomedical informatics , volume=. 2023 , publisher=
2023
-
[185]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[186]
First Conference on Language Modeling , year=
Gpqa: A graduate-level google-proof q&a benchmark , author=. First Conference on Language Modeling , year=
-
[187]
arXiv preprint arXiv:2506.17667 , year=
PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models , author=. arXiv preprint arXiv:2506.17667 , year=
-
[188]
arXiv preprint arXiv:2510.03185 , year=
PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning , author=. arXiv preprint arXiv:2510.03185 , year=
-
[189]
arXiv preprint arXiv:2501.13766 , year=
Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models , author=. arXiv preprint arXiv:2501.13766 , year=
-
[190]
arXiv preprint arXiv:2402.14008 , year=
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. arXiv preprint arXiv:2402.14008 , year=
-
[191]
arXiv preprint arXiv:2411.00836 , year=
Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models , author=. arXiv preprint arXiv:2411.00836 , year=
-
[192]
arXiv preprint arXiv:2509.16204 , year=
Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs , author=. arXiv preprint arXiv:2509.16204 , year=
-
[193]
arXiv preprint arXiv:2408.08302 , year=
Benchmarking the capabilities of large language models in transportation system engineering: Accuracy, consistency, and reasoning behaviors , author=. arXiv preprint arXiv:2408.08302 , year=
-
[194]
arXiv preprint arXiv:2404.03647 , year=
Capabilities of large language models in control engineering: A benchmark study on gpt-4, claude 3 opus, and gemini 1.0 ultra , author=. arXiv preprint arXiv:2404.03647 , year=
-
[195]
arXiv preprint arXiv:2310.15421 , year=
FANToM: A benchmark for stress-testing machine theory of mind in interactions , author=. arXiv preprint arXiv:2310.15421 , year=
-
[196]
arXiv preprint arXiv:2310.02255 , year=
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts , author=. arXiv preprint arXiv:2310.02255 , year=
-
[197]
arXiv preprint arXiv:1809.08887 , year=
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task , author=. arXiv preprint arXiv:1809.08887 , year=
-
[198]
arXiv preprint arXiv:2310.06770 , year=
Swe-bench: Can language models resolve real-world github issues? , author=. arXiv preprint arXiv:2310.06770 , year=
-
[199]
arXiv preprint arXiv:2107.03374 , year=
Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=
-
[200]
arXiv preprint arXiv:2405.16433 , year=
Cpsycoun: A report-based multi-turn dialogue reconstruction and evaluation framework for chinese psychological counseling , author=. arXiv preprint arXiv:2405.16433 , year=
-
[201]
Advances in neural information processing systems , volume=
Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[202]
European Conference on Computer Vision , pages=
Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? , author=. European Conference on Computer Vision , pages=. 2024 , organization=
2024
-
[203]
arXiv preprint arXiv:2502.00334 , year=
Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models , author=. arXiv preprint arXiv:2502.00334 , year=
-
[204]
arXiv preprint arXiv:1904.09675 , year=
Bertscore: Evaluating text generation with bert , author=. arXiv preprint arXiv:1904.09675 , year=
1904 arXiv
-
[205]
arXiv preprint arXiv:2110.14168 , year=
Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=
-
[206]
arXiv preprint arXiv:2501.18362 , year=
Medxpertqa: Benchmarking expert-level medical reasoning and understanding , author=. arXiv preprint arXiv:2501.18362 , year=
-
[207]
arXiv preprint arXiv:2307.16125 , year=
Seed-bench: Benchmarking multimodal llms with generative comprehension , author=. arXiv preprint arXiv:2307.16125 , year=
-
[208]
arXiv preprint arXiv:2309.14181 , year=
Q-bench: A benchmark for general-purpose foundation models on low-level vision , author=. arXiv preprint arXiv:2309.14181 , year=
-
[209]
Advances in Neural Information Processing Systems , volume=
Learn to explain: Multimodal reasoning via thought chains for science question answering , author=. Advances in Neural Information Processing Systems , volume=
-
[210]
Transactions on machine learning research , year=
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year=
-
[211]
Advances in neural information processing systems , volume=
Superglue: A stickier benchmark for general-purpose language understanding systems , author=. Advances in neural information processing systems , volume=
-
[212]
Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=
GLUE: A multi-task benchmark and analysis platform for natural language understanding , author=. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP , pages=
2018
-
[213]
Revisiting the evaluation of theory of mind through question answering , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages=
2019
-
[214]
arXiv preprint arXiv:2502.21017 , year=
PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues , author=. arXiv preprint arXiv:2502.21017 , year=
-
[215]
arXiv preprint arXiv:2501.13826 , year=
Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos , author=. arXiv preprint arXiv:2501.13826 , year=
-
[216]
arXiv preprint arXiv:2510.12712 , year=
Beyond seeing: Evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning , author=. arXiv preprint arXiv:2510.12712 , year=
-
[217]
arXiv preprint arXiv:2505.07671 , year=
Benchmarking retrieval-augmented generation for chemistry , author=. arXiv preprint arXiv:2505.07671 , year=
-
[218]
arXiv preprint arXiv:2502.09560 , year=
Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents , author=. arXiv preprint arXiv:2502.09560 , year=
-
[219]
Advances in Neural Information Processing Systems , volume=
Measuring multimodal mathematical reasoning with math-vision dataset , author=. Advances in Neural Information Processing Systems , volume=
-
[220]
arXiv preprint arXiv:2511.01163 , year=
ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation , author=. arXiv preprint arXiv:2511.01163 , year=
-
[221]
arXiv preprint arXiv:2508.17905 , year=
Pandora: Leveraging Code-driven Knowledge Transfer for Unified Structured Knowledge Reasoning , author=. arXiv preprint arXiv:2508.17905 , year=
-
[222]
arXiv preprint arXiv:2511.14159 , year=
MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs , author=. arXiv preprint arXiv:2511.14159 , year=
-
[223]
arXiv preprint arXiv:2406.05756 , year=
Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models , author=. arXiv preprint arXiv:2406.05756 , year=
-
[224]
arXiv preprint arXiv:2406.11303 , year=
Videovista: A versatile benchmark for video understanding and reasoning , author=. arXiv preprint arXiv:2406.11303 , year=
-
[225]
arXiv preprint arXiv:2505.21374 , year=
Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning? , author=. arXiv preprint arXiv:2505.21374 , year=
-
[226]
arXiv preprint arXiv:2509.01822 , year=
When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference , author=. arXiv preprint arXiv:2509.01822 , year=
-
[227]
Nature , volume=
Deepseek-r1 incentivizes reasoning in llms through reinforcement learning , author=. Nature , volume=. 2025 , publisher=
2025
-
[228]
arXiv preprint arXiv:2503.14190 , year=
Inferring Events from Time Series using Language Models , author=. arXiv preprint arXiv:2503.14190 , year=
-
[229]
ACM SigKDD Explorations Newsletter , volume=
Activity recognition using cell phone accelerometers , author=. ACM SigKDD Explorations Newsletter , volume=. 2011 , publisher=
2011
-
[230]
arXiv preprint arXiv:2506.20093 , year=
ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset , author=. arXiv preprint arXiv:2506.20093 , year=
-
[231]
arXiv preprint arXiv:2503.17452 , year=
CausalRivers--Scaling up benchmarking of causal discovery for real-world time-series , author=. arXiv preprint arXiv:2503.17452 , year=
-
[232]
Advances in Neural Information Processing Systems , volume=
Are language models actually useful for time series forecasting? , author=. Advances in Neural Information Processing Systems , volume=
-
[233]
arXiv preprint arXiv:2509.24803 , year=
TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models , author=. arXiv preprint arXiv:2509.24803 , year=
-
[234]
arXiv preprint arXiv:2507.06261 , year=
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=
-
[235]
arXiv preprint arXiv:2510.03519 , year=
TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoning , author=. arXiv preprint arXiv:2510.03519 , year=
-
[236]
arXiv preprint arXiv:2404.18824 , year=
Benchmarking benchmark leakage in large language models , author=. arXiv preprint arXiv:2404.18824 , year=
-
[237]
European heart journal , year=
Management of acute coronary syndromes in patients presenting without persistent ST-segment elevation , author=. European heart journal , year=
-
[238]
Journal of the American College of Cardiology , volume=
2018 ACC/AHA/HRS guideline on the evaluation and management of patients with bradycardia and cardiac conduction delay: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart Rhythm Society , author=. J...
2018
-
[239]
JAMA cardiology , volume=
Evaluation and treatment of patients with bradycardia and cardiac conduction delay: recommendations for permanent pacing , author=. JAMA cardiology , volume=. 2019 , publisher=
2019
-
[240]
Scientific data , volume=
PTB-XL, a large publicly available electrocardiography dataset , author=. Scientific data , volume=. 2020 , publisher=
2020
-
[241]
ISA annual convention , volume=
Gdelt: Global data on events, location, and tone, 1979--2012 , author=. ISA annual convention , volume=. 2013 , organization=
1979
-
[242]
Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V
CAMEF: Causal-augmented multi-modality event-driven financial forecasting by integrating time series patterns and salient macroeconomic announcements , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=
-
[243]
The Astrophysical Journal Supplement Series , volume=
LEAVES: an expandable light-curve data set for automatic classification of variable stars , author=. The Astrophysical Journal Supplement Series , volume=. 2024 , publisher=
2024
-
[244]
2009 international symposium on wearable computers , pages=
Potentials of enhanced context awareness in wearable assistants for Parkinson's disease patients with the freezing of gait syndrome , author=. 2009 international symposium on wearable computers , pages=. 2009 , organization=
2009
-
[245]
Anthropic PBC , title =
-
[246]
Natural Hazards Research , volume=
Flood prediction with time series data mining: Systematic review , author=. Natural Hazards Research , volume=. 2024 , publisher=
2024
-
[247]
IEEE Access , volume=
A comprehensive survey of deep transfer learning for anomaly detection in industrial time series: Methods, applications, and directions , author=. IEEE Access , volume=. 2024 , publisher=
2024
-
[248]
ACM Transactions on Management Information Systems , volume=
Time series prediction using deep learning methods in healthcare , author=. ACM Transactions on Management Information Systems , volume=. 2023 , publisher=
2023
-
[249]
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=
Fnspid: A comprehensive financial news dataset in time series , author=. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=
-
[250]
arXiv preprint arXiv:2408.13960 , year=
Time series analysis for education: Methods, applications, and future directions , author=. arXiv preprint arXiv:2408.13960 , year=
-
[251]
European Symposium on Artificial Neural Networks: Computational Intelligence and Machine Learning , pages=
Financial time series forecasting with machine learning techniques: A survey , author=. European Symposium on Artificial Neural Networks: Computational Intelligence and Machine Learning , pages=
-
[252]
Annals of Pharmacotherapy , volume=
Interrupted time series analysis in clinical research , author=. Annals of Pharmacotherapy , volume=. 2003 , publisher=
2003
-
[253]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Physics-guided foundation model for scientific discovery: An application to aquatic science , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[254]
2025 , eprint=
MiMo-VL Technical Report , author=. 2025 , eprint=
2025
-
[255]
arXiv preprint arXiv:2503.19786 , year=
Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=
-
[256]
5: Advancing open-source multimodal models in versatility, reasoning, and efficiency , author=
Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency , author=. arXiv preprint arXiv:2508.18265 , year=
-
[257]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Timecap: Learning to contextualize, augment, and predict time series events with large language model agents , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[258]
arXiv preprint arXiv:2410.18959 , year=
Context is key: A benchmark for forecasting with essential textual information , author=. arXiv preprint arXiv:2410.18959 , year=
-
[259]
arXiv preprint arXiv:2601.18744 , year=
TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models , author=. arXiv preprint arXiv:2601.18744 , year=
-
[260]
arXiv preprint arXiv:2510.03255 , year=
SciTS: Scientific Time Series Understanding and Generation with LLMs , author=. arXiv preprint arXiv:2510.03255 , year=
-
[261]
arXiv preprint arXiv:2506.10412 , year=
Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series , author=. arXiv preprint arXiv:2506.10412 , year=
-
[262]
Advances in Neural Information Processing Systems , volume=
Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models , author=. Advances in Neural Information Processing Systems , volume=
-
[263]
arXiv preprint arXiv:2601.06559 , year=
ArrowGEV: Grounding Events in Video via Learning the Arrow of Time , author=. arXiv preprint arXiv:2601.06559 , year=
-
[264]
arXiv preprint arXiv:2303.11381 , year=
Mm-react: Prompting chatgpt for multimodal reasoning and action , author=. arXiv preprint arXiv:2303.11381 , year=
-
[265]
arXiv preprint arXiv:2305.11860 , year=
Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs , author=. arXiv preprint arXiv:2305.11860 , year=
-
[266]
arXiv preprint arXiv:2305.11738 , year=
Critic: Large language models can self-correct with tool-interactive critiquing , author=. arXiv preprint arXiv:2305.11738 , year=
-
[267]
arXiv preprint arXiv:2503.06749 , year=
Vision-r1: Incentivizing reasoning capability in multimodal large language models , author=. arXiv preprint arXiv:2503.06749 , year=
-
[268]
Aha Moment
R1-Zero's" Aha Moment" in Visual Reasoning on a 2B Non-SFT Model , author=. arXiv preprint arXiv:2503.05132 , year=
-
[269]
arXiv preprint arXiv:2502.14768 , year=
Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning , author=. arXiv preprint arXiv:2502.14768 , year=
-
[270]
Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples , author=
-
[271]
Findings of the Association for Computational Linguistics: ACL 2024 , pages=
Everything of thoughts: Defying the law of penrose triangle for thought generation , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=
2024
-
[272]
Proceedings of the AAAI conference on artificial intelligence , volume=
Graph of thoughts: Solving elaborate problems with large language models , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[273]
arXiv preprint arXiv:2308.04371 , year=
Cumulative reasoning with large language models , author=. arXiv preprint arXiv:2308.04371 , year=
-
[274]
arXiv preprint arXiv:2404.17140 , year=
Small language models need strong verifiers to self-correct reasoning , author=. arXiv preprint arXiv:2404.17140 , year=
-
[275]
Proceedings of the 32nd ACM International Conference on Multimedia , pages=
Cantor: Inspiring multimodal chain-of-thought of mllm , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=
-
[276]
arXiv preprint arXiv:2203.11171 , year=
Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=
-
[277]
arXiv preprint arXiv:2510.14901 , year=
Reasoning with sampling: Your base model is smarter than you think , author=. arXiv preprint arXiv:2510.14901 , year=
-
[278]
arXiv preprint arXiv:2408.03314 , year=
Scaling llm test-time compute optimally can be more effective than scaling model parameters , author=. arXiv preprint arXiv:2408.03314 , year=
-
[279]
arXiv preprint arXiv:2407.21787 , year=
Large language monkeys: Scaling inference compute with repeated sampling , author=. arXiv preprint arXiv:2407.21787 , year=
-
[280]
arXiv preprint arXiv:2302.00923 , year=
Multimodal chain-of-thought reasoning in language models , author=. arXiv preprint arXiv:2302.00923 , year=
-
[281]
arXiv preprint arXiv:2205.10625 , year=
Least-to-most prompting enables complex reasoning in large language models , author=. arXiv preprint arXiv:2205.10625 , year=
-
[282]
arXiv preprint arXiv:2210.02406 , year=
Decomposed prompting: A modular approach for solving complex tasks , author=. arXiv preprint arXiv:2210.02406 , year=
-
[283]
European heart journal , volume=
2020 ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: The Task Force for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation of the European Socie...
2020
-
[284]
arXiv preprint arXiv:2503.01875 , year=
Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement , author=. arXiv preprint arXiv:2503.01875 , year=
-
[285]
arXiv preprint arXiv:2111.09543 , year=
Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing , author=. arXiv preprint arXiv:2111.09543 , year=
-
[286]
Proceedings of the 29th symposium on operating systems principles , pages=
Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=
-
[287]
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations , pages=
Ragas: Automated evaluation of retrieval augmented generation , author=. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations , pages=
-
[288]
arXiv preprint arXiv:2502.07608 , year=
Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Prompting , author=. arXiv preprint arXiv:2502.07608 , year=
-
[289]
arXiv preprint arXiv:2406.18665 , year=
Routellm: Learning to route llms with preference data , author=. arXiv preprint arXiv:2406.18665 , year=
-
[290]
Advances in Neural Information Processing Systems , volume=
Routerdc: Query-based router by dual contrastive learning for assembling large language models , author=. Advances in Neural Information Processing Systems , volume=
-
[291]
The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
-
[292]
Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1...
-
[293]
arXiv preprint arXiv:2607.08940 , year=
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning , author=. arXiv preprint arXiv:2607.08940 , year=
-
[294]
Advances in Neural Information Processing Systems , volume=
Chatqa: Surpassing gpt-4 on conversational qa and rag , author=. Advances in Neural Information Processing Systems , volume=
-
[295]
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
Routing to the expert: Efficient reward-guided ensemble of large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2024
-
[296]
arXiv preprint arXiv:2411.19146 , year=
Puzzle: Distillation-based nas for inference-optimized llms , author=. arXiv preprint arXiv:2411.19146 , year=
-
[297]
arXiv preprint arXiv:2408.00118 , year=
Gemma 2: Improving open language models at a practical size , author=. arXiv preprint arXiv:2408.00118 , year=
-
[298]
arXiv preprint arXiv:2410.03834 , year=
Graphrouter: A graph-based router for llm selections , author=. arXiv preprint arXiv:2410.03834 , year=
-
[299]
arXiv preprint arXiv:2410.10813 , year=
Longmemeval: Benchmarking chat assistants on long-term interactive memory , author=. arXiv preprint arXiv:2410.10813 , year=
-
[300]
arXiv preprint arXiv:2108.07732 , year=
Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.