{"id":"dd650591-b1d4-4715-8853-f458065f081c","arxiv_id":"2412.10823","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"Using BERTopic-based news clustering, daily price granularity, and instruction tuning, the authors report 63% binary accuracy for weekly stock movement prediction, up from a 55% baseline.","lead":"This paper proposes feeding an LLM a mix of clustered news, daily price changes, and explicit instructions so it can better predict weekly stock movements. The authors report that directional prediction accuracy rises from 55% to 63% on 380 company-week observations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 8% accuracy gain is unverifiable without a disclosed train/test split: if the 380 evaluation observations overlap with instruction-tuning instances whose GPT-4o targets were generated from known future movements, the comparison is a leakage artifact rather than evidence.","rationale":"The reader's weakest_assumption identified both the unvalidated centroid/cluster-size assumptions and the underspecified evaluation procedure. I concentrate on the evaluation split because it is the condition that makes the headline 8% claim meaningful at all. Even if the clustering assumptions are plausible, the reported numerical gain is only interpretable if the comparison is out-of-sample; with 380 observations spanning 20 companies and no stated split, the risk is that the fine-tuning set and the evaluation set are the same weeks or contain the same GPT-4o-generated target analyses. The paper's own description of training data generation makes this concern concrete: the GPT-4o targets are produced using known future movements, so any overlap between those targets and the evaluation labels would let the model indirectly see the answer. The proposed test, releasing the exact row-level split and recomputing on a strictly disjoint holdout, would settle the matter. My recommendation is unchanged because the reader already rejected the paper on these grounds, and the missing split is a correctness-critical omission that supports rejection at moderate confidence.","tokens_in":7155,"tokens_out":3871,"duration_ms":37109,"concrete_test":"Obtain from the authors the complete data construction script and the exact observation index, company identifier, and week for each of the 380 rows; verify that no week used for fine-tuning overlaps with any evaluation week and no GPT-4o target used in fine-tuning is derived from the same week's future return as an evaluation prompt. Then recompute Table 1 on the disjoint holdout subset only, reporting per-company accuracy and a paired McNemar test. If the split is already non-overlapping, releasing it would resolve the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 63.0% versus 55.0% binary accuracy comparison in Table 1. The paper states that the instruction-tuning dataset pairs prompts with GPT-4o-generated analyses 'based on known future movements' and that Llama3-8B is fine-tuned on this dataset, but it never specifies the train/test split. Appendix C reports a 'test period' (2024/5/26-2024/9/29) for Boeing only; no corresponding training period or evaluation dates for the other 19 companies are given. If the 380 evaluation observations overlap with the instruction-tuning instances, or with the GPT-4o targets derived from those same weeks' future returns, then the model has seen ground-truth-derived target text during training, and the 8 percentage point gap is an artifact of memorization rather than a property of dissemination-aware prompting. The comparison also lacks per-company or per-week breakdowns and significance testing, so even a correct split would not fully establish the claim; but the split is the load-bearing condition because it determines whether the experiment is an out-of-sample evaluation at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinGPT, a data-centric pipeline for sentiment-based weekly stock movement prediction. The method increases stock price granularity from weekly to daily, clusters weekly news with BERTopic, enriches LLM prompts with cluster metadata (cluster size and temporal span) and temporal instructions, and constructs an instruction-tuning dataset with GPT-4o-generated analyses. Llama3-8B is fine-tuned on this dataset, and the authors report binary accuracy and ROUGE scores, claiming an 8% accuracy improvement over a FinRobot-based baseline (63.0% vs. 55.0% on 380 observations across 20 companies). A Boeing case study is used as additional qualitative evidence.","tokens_in":7474,"tokens_out":5953,"duration_ms":53854,"significance":"If the empirical claim were validated, the idea of making LLM prompts dissemination-aware through news clustering would be a useful and broadly applicable contribution to sentiment-based financial prediction. The paper's strengths include a clearly described data-processing pipeline, explicit prompt templates in the appendix, and a training-parameter table that aids reproducibility. The central result, however, is currently not established: the evaluation lacks a described train/test split, uncertainty quantification, external baselines, and validation of the two assumptions on which the HG-NC improvement rests. These gaps are load-bearing because the headline 8% gain is the paper's main empirical contribution.","major_comments":[{"comment":"The manuscript never specifies how the 380 evaluation observations are separated from the instruction-tuning data. Because the GPT-4o targets are generated \"based on known future movements,\" any temporal or company overlap between training and test instances would let the fine-tuned Llama3-8B memorize answers and would make the 63.0% versus 55.0% gap a leakage artifact. Please report the training period, test period, company overlap, and the exact construction of the 380-instance test set, and verify that all test prompts are disjoint from the instruction-tuning prompts.","section":"§4.4 and §5 (Table 1)"},{"comment":"No uncertainty quantification is provided for the headline accuracies. With 380 binary observations, the 8-percentage-point difference needs a paired significance test (e.g., McNemar's test) and per-company/per-week breakdowns; the word \"significant\" in the table caption is not justified by the evidence presented.","section":"§5, Table 1"},{"comment":"The abstract and conclusion claim improvement over \"existing methods,\" but the only comparator in Table 1 is the authors' own FinRobot data format used as the baseline. No existing published model or standard stock-movement benchmark is evaluated. Either add external baselines (e.g., FinBERT, LLMFactor, or a news-only LLM baseline) or restrict the claim to \"improves the FinRobot prompt format.\"","section":"§1 and §5"},{"comment":"The two stated assumptions—that the centroid article is informationally sufficient and that cluster size indicates market impact—are load-bearing for the HG-NC result but are not validated. I recommend an ablation that compares (a) centroid article only, (b) centroid plus cluster metadata, and (c) a random or full-cluster sample, along with a direct correlation analysis between cluster size and subsequent weekly return.","section":"§4.2"},{"comment":"The Boeing case study is presented as supporting evidence, but it is a post hoc analysis of a single company, the \"strong correlation\" in Figure 2 is not quantified, and the case definitions are internally inconsistent (the figure caption says \"Case 1: HG-NC correct vs. NC incorrect,\" where NC should presumably be HG). This section should be either removed or reframed as an illustrative example with proper statistics.","section":"§5, Case Study"}],"minor_comments":[{"comment":"The table header is corrupted (\"Method A vg Acc Long term Short term\"); the column names need to be cleaned.","section":"Table 1"},{"comment":"Table 4 uses \"News Ration\" instead of \"News Ratio,\" and the text alternates between \"high-coherence\" and \"high-cohesion\"; please standardize terminology.","section":"Table 4 and §4.2"},{"comment":"The model name appears as \"LLAMA-3-8B\" in Appendix A and \"Llama3-8B\" elsewhere; use one consistent spelling.","section":"Appendix A and §4.4"},{"comment":"The reference list contains malformed citation keys such as \"elahi2024combining\" and \"F ingpt\"; the bibliography needs cleanup.","section":"References"},{"comment":"For a data-centric paper, a data/code availability statement and release of the instruction-tuning dataset with explicit splits would materially improve reproducibility.","section":"Reproducibility"},{"comment":"The ROUGE evaluation compares generated reasoning to GPT-4o-generated references, not to human ground truth; the text should say \"reference\" rather than \"ground-truth sentiment-based analysis.\"","section":"§5, ROUGE Scores"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently unverifiable because of the missing train/test split. I would ask the authors to provide the split and re-run the evaluation if needed; if they cannot, the paper should not be published. The self-citation pattern in Related Work is also worth editorial attention, though I do not treat it as a technical flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clearly written data-preparation study from the FinGPT group. The new piece is a concrete pipeline: daily price granularity plus BERTopic news clustering, using cluster size and temporal span as dissemination features, fed into instruction-tuned Llama-3. The description is specific enough to reimplement, and the authors state their two assumptions (centroid article is informationally sufficient; cluster size proxies market impact) rather than hiding them. That is to their credit.\n\nThe evaluation does not support the headline. The paper reports 55.0% baseline vs 63.0% HG-NC binary accuracy on 380 company-weeks, but it never states the train/test split. The training targets are GPT-4o analyses 'based on known future movements,' and if the 380 evaluation observations overlap with those instruction-tuning instances, the gain is memorization, not generalization. The Boeing test period (5/26–9/29, 2024) is given in Appendix C, but there is no corresponding training period or split for the other 19 companies. This is the load-bearing gap, not a minor omission. There are also no significance tests, no confidence intervals, no per-company breakdowns, and no sensitivity analysis for the clustering thresholds (cohesion 0.6, topic caps, etc.).\n\nThe secondary evidence is weak in predictable ways. The rising 'long-term'/'short-term' word frequencies are instruction-following, not accuracy. The Boeing case study is post hoc; the Figure 2 case classification is a descriptive pattern. ROUGE gains are small (0.450 to 0.472 ROUGE-1) and ROUGE is a weak proxy for reasoning.\n\nThe baseline is the authors' own FinRobot-style prompt format, so the 8% is relative to their own prior format, not to an external method. That does not make the pipeline useless, but it changes what the claim means.\n\nWho gets value: practitioners building LLM trading pipelines might borrow the prompt and clustering recipe. As a scientific claim, the 8% is unverified until the split, artifacts, and statistics arrive.\n\nRecommendation: send to peer review, but make the train/test split and per-period results a condition. The idea deserves referee time; the current evidence does not yet carry the claim.","headline":"A clearly written data-prep recipe for LLM stock prediction whose headline 8% gain is not yet supported because the paper never discloses the train/test split.","tokens_in":7959,"tokens_out":3025,"would_cite":false,"duration_ms":26457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding news-dissemination data to LLM prompts lifts stock-movement accuracy from 55% to 63%.","keywords":["financial sentiment analysis","stock movement prediction","large language models","instruction tuning","news clustering","BERTopic","dissemination-aware","context-enriched prompts"],"falsifier":"Replace each centroid article with a randomly chosen article from the same cluster and re-measure binary accuracy on the same 380 observations; if accuracy does not drop, the centroid assumption is not the carrier. Likewise, shuffle the cluster-size and temporal-span metadata while keeping the article text fixed; if the 63% result persists, the dissemination signal is not doing the work.","tokens_in":7000,"feed_emoji":"📈","tokens_out":5688,"duration_ms":48789,"temperature":0.7,"pith_summary":"This paper argues that stock-movement prediction from financial news improves when the prompt given to an instruction-tuned language model includes not just article text but also how the news spread and what the price data looked like day by day. The proposed pipeline first enriches stock-price context with daily closing prices and returns, then clusters weekly news into topics and keeps one representative article per cluster together with the cluster's size and temporal span. These enriched prompts are used to fine-tune a Llama3-8B model, and the resulting system predicts weekly up/down moves with 63.0% binary accuracy versus 55.0% for the baseline on 380 observations spanning 20 companies. The authors see the 8-percentage-point gain as evidence that dissemination breadth and explicit temporal instructions help LLMs read financial news.","feed_headline":"News-dissemination data lifts stock-movement accuracy to 63%","feed_subtitle":"Adding daily prices and news cluster metadata to LLM prompts beats the 55% baseline by 8 points.","key_machinery":"The load-bearing object is the HG-NC prompt format, a data-processing and prompt-engineering pipeline. In the HG step, weekly aggregate price trends are replaced by a table of daily closing prices and daily returns, giving the model an explicit timeline. In the NC step, BERTopic clusters the week's news for a company; for clusters whose average pairwise similarity exceeds 0.6 the pipeline selects the article closest to the centroid and records cluster size and temporal span, while low-cohesion clusters are capped at two articles. The prompt then instructs the model to read the centroid article as the topic's most complete statement, to treat cluster size as a proxy for market impact, and to separate short-term from long-term news effects. This instruction-tuning dataset, generated with GPT-4o outputs and used to fine-tune Llama3-8B, is what carries the reported accuracy gain.","core_discovery":"The paper's central claim is a data-format effect: representing a week's news as a small set of topic clusters, each reduced to its centroid article plus two dissemination numbers (how many articles formed the cluster and over what time span), and pairing this with daily rather than weekly price data, moves an instruction-tuned LLM's weekly directional accuracy from 55.0% to 63.0%. The paper attributes the jump to two mechanisms: daily price granularity lets the model align news events with same-day market reactions, and cluster metadata quantifies a topic's reach. Reasoning output also shifts, with the words 'long-term' and 'short-term' appearing far more often, and ROUGE scores for the prediction-analysis section rise, which the paper takes as evidence that the model is weighing temporal and dissemination factors rather than merely fitting text.","pith_inferences":["A reader can test whether the centroid assumption is the active ingredient by replacing each centroid article with a random article from the same cluster and re-measuring accuracy; if the gain survives, dissemination metadata alone, not article choice, is doing the work.","The reported comparison treats all 380 observations as a single pool. A natural next step, not in the paper, is to report per-company and per-week accuracy with confidence intervals, because a few clustered weeks or companies could drive the difference.","The clustering-ratio correlation in the Boeing case suggests a broader testable law: other companies should show a positive relationship between high-cohesion news share and the HG-NC advantage.","Because the prompts are the only thing changed between conditions, the design isolates a format effect; one can extend it to non-text LLM data by encoding dissemination as structured features rather than prose."],"forward_implications":["If the 8-point gain is real, news dissemination metadata is a usable signal for LLM-based financial forecasting, not just the sentiment polarity of individual articles.","Daily price granularity supports a distinction between short- and long-term news effects, and the model's increased use of those terms suggests the instruction is being followed.","Because the method is data-format driven and model-independent, the same clustering-and-granularity treatment can be layered onto other financial LLMs and prompt-based predictors.","On Boeing, weeks with a high ratio of articles in high-cohesion clusters tended to be weeks where HG-NC beat HG, suggesting the clustering quality itself predicts when the method will help.","ROUGE improvements imply the enriched prompts produce more comprehensive reasoning, not just better binary guesses."],"supporting_citations":[{"why":"Supplies BERTopic, the topic-modeling tool that creates the news clusters and centroid representatives used in the NC step.","marker":"Grootendorst 2022"},{"why":"Defines the standard instruction-tuning framework and baseline data organization that the paper enriches with daily prices and clustering.","marker":"Yang et al. 2024"},{"why":"Provides the FinGPT instruction-tuning benchmark that the training-data construction builds on.","marker":"Wang, Yang, and Wang 2023"},{"why":"Establishes the FinGPT open-source financial LLM line that the proposed method extends.","marker":"Yang, Liu, and Wang 2023"},{"why":"Supplies the ROUGE metrics used to compare reasoning quality across baseline, HG, and HG-NC.","marker":"Lin 2004"}],"fun_headline_variants":["News spread data in LLM prompts lifts stock prediction to 63%","FinGPT boosts stock-move accuracy 8 points with news clusters","Dissemination-aware LLM hits 63% on stock direction calls","Context and reach data sharpen LLM stock predictions to 63%","Adding news impact metrics to LLMs improves stock calls by 8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that a cluster's centroid article is informationally sufficient and that cluster size tracks market impact, with no independent check of either.","fun_headline_variants_meta":{"raw":{"variants":["News spread data in LLM prompts lifts stock prediction to 63%","FinGPT boosts stock-move accuracy 8 points with news clusters","Dissemination-aware LLM hits 63% on stock direction calls","Context and reach data sharpen LLM stock predictions to 63%","Adding news impact metrics to LLMs improves stock calls by 8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1551,"prompt_tokens":886,"completion_tokens":665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":502,"tokens_out":665,"duration_ms":6429,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:34:20.440287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace each centroid article with a randomly chosen article from the same cluster and re-measure binary accuracy on the same 380 observations; if accuracy does not drop, the centroid assumption is not the carrier. Likewise, shuffle the cluster-size and temporal-span metadata while keeping the article text fixed; if the 63% result persists, the dissemination signal is not doing the work.","supporting_citations":[],"review_version":1}