Pith. sign in

REVIEW 6 cited by

Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.06735 v2 pith:JPMPJTU4 submitted 2024-11-11 cs.AI

Kai Kim , Howard Tsai , Rajat Sen , Abhimanyu Das , Zihao Zhou , Abhishek Tanpure , Mathew Luo , Rose Yu This is my paper
classification cs.AI
keywords datamultimodalforecastingtimedatasetseriestextavailable
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    Uncontrolled text–time-series fusion underperforms unimodal baselines; constrained fusion and a low-rank Controlled Fusion Adapter recover gains without changing the TS backbone.

  2. Text Reinforcement for Multimodal Time Series Forecasting

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Reinforcement learning trains an LLM to generate improved text from time series, improving multimodal forecasting on Time-MMD.

  3. TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Reinforcement learning with a composite reward lifts Qwen2.5-VL-3B to 75.29% average accuracy on TIMERBED, above prompt-based GPT-4o and classical time-series baselines.

  4. A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models

    cs.AI 2025-09 conditional novelty 5.0 of 10

    The authors organize LLM-based time series reasoning into three exclusive topologies (direct, chain, branch) crossed with four objectives, and use them to label 125 papers, benchmarks, and resources.

  5. XFMNet: Decoding Cross-Site and Nonstationary Water Patterns via Stepwise Multimodal Fusion for Long-Term Water Quality Forecasting

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    XFMNet fuses local water-quality time series with remote-sensing precipitation imagery through stepwise multimodal fusion to improve long-term, multi-site water quality forecasting.

  6. Diffusion Models for Time Series Forecasting: A Survey

    stat.ML 2025-07 conditional novelty 4.0 of 10

    A survey classifies diffusion-based time series forecasting models into a two-axis taxonomy by conditioning source and integration method.

Pith tools