{"id":"d0004886-c097-4303-beaa-bb4e56055bbe","arxiv_id":"2412.03148","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new fine-grained benchmark and an observation-and-memory chain-of-thought fine-tuning method for simulating social media user behavior.","lead":"This paper introduces FineRob, a dataset of 78.6k multiple-choice questions that tests whether large language models can predict a social media user's next behavior, broken into object, type, and content. It also proposes OM-CoT, a fine-tuning method that improves this prediction by making the model explicitly reason about current observations and past memories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OM-CoT's measured gains may be an artifact of FineRob's similarity-sampled distractors (Sec. 3.2); without a retrieval baseline or distractor-robustness check, the central claim lacks support.","rationale":"The reader's weakest assumption is that FineRob's automatically generated labels and distractors may be invalid. I agree that this is a key risk, but I want to sharpen it: the specific construction of distractors (Sec. 3.2) likely creates a shortcut that aligns with the very mechanism OM-CoT is trained to use. Even if every label is correct, the multiple-choice format may be solvable by nearest-neighbor retrieval in embedding space, especially for object and content where distractors are deliberately similar to the correct answer. OM-CoT's <MEM> token explicitly encourages the model to compare observed options with historical behavior, making it well-suited to exploit such a shortcut. The ablation in Table 5 shows that removing history causes a steep drop, which is consistent with this interpretation. The central claim is not that OM-CoT is a better reasoner per se, but that it improves behavior simulation accuracy; if the benchmark rewards retrieval rather than reasoning, the measured improvement does not support the claim. My proposed check—a retrieval baseline using the same encoder—directly tests whether the benchmark requires more than surface similarity. This is a concrete, falsifiable test that would either confirm or allay the concern. I am not asserting that the authors are gaming the benchmark; I am pointing out a risk that the paper currently does not address. Because the reader's verdict is already conditional, my analysis does not change it: the paper should be accepted only after such a validation is provided. I therefore keep the verdict unchanged, with 'partial' agreement to note that my concern is a more specific mechanism than the reader's general label-validity worry.","tokens_in":124,"tokens_out":4470,"duration_ms":56107,"concrete_test":"Run a retrieval-only baseline on FineRob's test set: for each multiple-choice question, score each option by mean cosine similarity (using the same m3e encoder as Sec. 3.2) to the user's provided behavior history, and pick the highest-scoring option. Report F1 for object and content sub-tasks on all three platforms. If this baseline matches or exceeds the zero-shot and Std-CoT numbers, and comes within a few points of OM-CoT, the benchmark is gamed by surface similarity and the central claim loses credibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FineRob's negative options for object and content are selected by sentence-embedding similarity to the correct option with sentiment alignment (Sec. 3.2). This creates a test distribution in which the correct answer is often the candidate most similar to the user's past behavior—exactly the signal OM-CoT's <MEM> token is trained to exploit (Sec. 4.2, Table 5). The oracle CoT generation (Sec. 4.2) reveals the correct answer to the teacher, so the SFT data may teach a shallow rule linking history similarity to the answer, rather than a generalizable reasoning pattern. The central claim that OM-CoT improves behavior simulation accuracy therefore depends on the assumption that FineRob requires genuine reasoning. That assumption is untested: no retrieval baseline is reported, and no ablation varies the distractor construction. If OM-CoT's advantage over Std-CoT and zero-shot is mostly retrieval of the most similar past item, it would not constitute behavior simulation ability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FineRob, a new multilingual benchmark for fine-grained behavior simulation on social media, in which each user behavior is decomposed into three elements (object, type, content) and converted into multiple-choice QA records derived from the real timelines of 1,866 users on Twitter, Reddit, and Zhihu. The authors analyze zero-shot CoT reasoning and identify two patterns ('role stereotype' and 'observation and memory'), then propose OM-CoT, a fine-tuning method that inserts special tokens <ANA> and <MEM> to structure reasoning into observation analysis and memory analysis. Experiments with nine LLMs and fine-tuning on two open-source models report consistent gains for OM-CoT over zero-shot, few-shot, and standard CoT fine-tuning, along with ablations on prompt components, history size, and the two special tokens.","tokens_in":11842,"tokens_out":4048,"duration_ms":38842,"significance":"If the benchmark and the method hold up, this is a useful contribution: FineRob is one of the first real-user, multi-platform, fine-grained behavior simulation datasets with a disjoint-user train/test split, and OM-CoT is a simple, reproducible fine-tuning recipe that shows consistent improvements across models and sub-tasks. The authors release code and data, and the ablations (Tables 5 and 6) give insight into which prompt components matter. However, the central claim that OM-CoT improves genuine behavior simulation depends on two untested assumptions: that the automatically constructed multiple-choice distractors are not gameable by similarity matching, and that the oracle CoT used for training does not leak the answer. The reported zero standard deviations and the test-set-based choice of history window further weaken the empirical case.","major_comments":[{"comment":"The distractor construction for object and content options selects negatives by sentence-embedding similarity to the correct answer, with sentiment alignment. This creates a test distribution in which the correct option may be the candidate most similar to the user's past behaviors, exactly the signal that OM-CoT's <MEM> token is trained to exploit (Section 4.2, Table 5). No retrieval baseline is reported, and no ablation varies the distractor construction. If a simple history-similarity retriever already achieves high accuracy, or if OM-CoT's advantage shrinks when distractors are randomly sampled, the claimed behavior-simulation ability would not be established. Please add a retrieval baseline (e.g., selecting the option with maximum embedding similarity to the user's recent history) and a distractor-robustness check (e.g., random distractors vs. current similar-sentiment distractors) to show that the benchmark requires reasoning beyond similarity matching.","section":"Section 3.2"},{"comment":"Oracle CoT generation includes the correct answer in the input prompt, and the paper states that the prompt was 'carefully adjusted' to avoid revealing the answer, but no verification is provided. If the generated CoT text contains answer-specific hints (e.g., restating the correct option's wording), the SFT training would teach the model to reproduce a pattern that is correlated with the answer, and OM-CoT's gains over Std-CoT could reflect this leakage rather than a generalizable reasoning skill. Please quantify leakage (e.g., train a classifier to predict the answer from the CoT text alone, or manually audit a random sample) and, ideally, validate on a fresh set of users whose oracle CoTs are generated without any answer information.","section":"Section 4.2"},{"comment":"Many scores are reported with zero standard deviation (e.g., Table 4: Mistral-7b std-CoT-FT Object 31.56±0.0, Type 31.40±0.0, Twitter Object 55.76±0.0; Solar-10.7b OM-CoT-FT several entries ±0.0). Since inference uses sampling with temperature 0.1 and results are averaged over three trials, exact zeros across 17.6k test items are surprising and undermine confidence in the reported differences. Please clarify whether decoding is actually deterministic for these entries or provide the full distribution (e.g., per-trial scores or confidence intervals) so that the significance of the OM-CoT gains can be assessed.","section":"Tables 3 and 4"},{"comment":"The history window size (30) is chosen after evaluating performance on the Twitter test set across sizes from 10 to all entries, and the main results in Table 4 use this chosen value on the same test set. This is test-set tuning and can inflate the reported performance. Please hold out a validation split for selecting the history window and other hyperparameters, or report the sensitivity of the Table 4 conclusions to the window size.","section":"Section 5.3, Figure 5"}],"minor_comments":[{"comment":"There are several typos and grammatical issues, e.g., 'simulat' in the prompt description, 'The last two models is specialize for Chinese context', and 'Mistrail-7b' in Table 5. A careful proofread is needed.","section":"Section 5.1"},{"comment":"The paper says 'randomly sample three options' for distractors, but does not state the total number of options per question (presumably four). Please make the option count explicit for each behavior element.","section":"Section 3.2"},{"comment":"The quantitative analysis in Figure 4 uses similarity between CoT reasoning text and prompt parts, but the methodology (e.g., which similarity measure, how the similarity intervals are defined) is not described. Please clarify so the reader can interpret the claimed correlation.","section":"Section 4.1"},{"comment":"The behavior type taxonomy is central to the dataset, but the table does not indicate the number of instances per type or how types are balanced across platforms. A brief distributional summary would help assess potential biases.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The reviewer's stress-test concern about the retrieval baseline is well founded and should be addressed by the authors. The paper's central claim depends on FineRob measuring something beyond similarity matching, and the current evidence does not rule out a shallow retrieval shortcut. I would not reject at this stage, because the dataset and method are potentially valuable, but the requested experiments (retrieval baseline, distractor robustness, leakage audit, validation split) are necessary before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FineRob is a genuinely useful new asset: 78.6k multiple-choice QA records built from real timelines of 1,866 users on three platforms, split by user so the test set is not contaminated. The object/type/content decomposition is a sensible annotation scheme, and the OM-CoT special-token method (<ANA>, <MEM>) is a clean extension of CoT fine-tuning. The paper also ships code and data, which helps reproducibility. The main experiments are standard, and OM-CoT beats both zero-shot and standard-CoT fine-tuning on most subtasks, so the structure of the reasoning instruction is doing something.\n\nThe soft spots are real but not fatal. The biggest one: distractors for object and content are picked by sentence-embedding similarity to the correct answer with sentiment alignment. That means the correct option is often the one most similar to the user's past behavior—exactly the signal the <MEM> token is trained to exploit. The paper offers no retrieval baseline and no distractor-robustness ablation. A cheap control would be to choose the candidate with max average similarity to the history and see what accuracy that gets. Until that is reported, part of OM-CoT's gain could be shallow retrieval rather than behavior simulation. The stress-test note gets this right, though I wouldn't call the claim unsupported: the std-CoT-FT comparison and the <ANA>/<MEM> ablations show the method is doing more than memorizing.\n\nSecond, the dataset has no human validation. Labels and distractors are auto-generated; noisy labels would compress all model differences. The paper should include a small human agreement study.\n\nThird, the variance reporting is unconvincing. Tables 4 and 5 show many 0.00 standard deviations, which is hard to believe even at low temperature. Report per-run numbers.\n\nFourth, full real-user timelines are a privacy risk. The ethics section is one paragraph and doesn't address consent, anonymization, or the possibility of re-identification.\n\nMinor: the reasoning-pattern taxonomy comes from the authors' own analysis on the same benchmark used to motivate and test the method; there's some circularity, and an independent annotation of reasoning patterns would strengthen it.\n\nWho's this for: people building social-media agents, personalization, and bot detection. The benchmark will probably be used even if the method claim is later refined. I'd send it to review, but I'd ask the authors to add a retrieval baseline, a human validation sample, and proper variance reporting before acceptance.","headline":"Useful new benchmark with a real confound in its distractor design; worth reviewing, but the central claim needs a retrieval baseline before I'd trust it.","tokens_in":12304,"tokens_out":2752,"would_cite":true,"duration_ms":25728,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs simulate social-media users more accurately when their reasoning is explicitly split into observation and memory analysis.","keywords":["large language models","role-playing","behavior simulation","social media","chain-of-thought fine-tuning","OM-CoT","FineRob","personalization"],"falsifier":"Take a random sample of FineRob test items and have human raters decide whether the recorded behavior is what the user would most plausibly do; if agreement with the automatic labels is low, the benchmark scores are not measuring real behavior simulation. Alternatively, fine-tune with the CoT split at random positions into two labeled segments instead of the true <ANA>/<MEM> split; if accuracy does not drop, the semantic roles of the tokens are not the cause.","tokens_in":11409,"feed_emoji":"🎭","tokens_out":4749,"duration_ms":40632,"temperature":0.7,"pith_summary":"The paper tries to establish that fine-grained user behavior simulation is a measurable, learnable task and that an LLM's success at it depends on how it reasons. It introduces FineRob, a benchmark of 78.6k multiple-choice questions built from the real behavioral histories of 1,866 users on Twitter, Reddit, and Zhihu, with each action decomposed into object, type, and content. Analyzing chain-of-thought traces, the authors find that models usually reason in one of two ways: relying on role stereotypes or comparing current options against remembered past behaviors, with the latter producing more accurate predictions. Based on this, they propose OM-CoT fine-tuning, which forces the model to write separate observation analyses and memory analyses using special tokens. On Mistral-7B and Solar-10.7B, this structured fine-tuning outperforms zero-shot, few-shot, and standard CoT fine-tuning across most sub-tasks, so the central claim is that structuring reasoning into observation and memory is a practical way to improve behavior simulation.","feed_headline":"LLMs predict user actions better when told to observe, then remember","feed_subtitle":"On a new 78.6k-question social-media benchmark, observation-plus-memory reasoning beats plain chain-of-thought.","key_machinery":"The load-bearing mechanism is OM-CoT fine-tuning, a supervised fine-tuning procedure whose training data is a chain-of-thought trace reorganized with two special tokens: <ANA> wraps analysis of the candidate options at hand, and <MEM> wraps recall of the user's historical behaviors, followed by an explicit final decision. An oracle setting generates the CoT with the correct answer visible so the teacher can reference the real behavior without leaking the answer in the reasoning text. FineRob is the measuring instrument: each behavior is split into object, type, and content, and each question is a multiple-choice item whose negative options are sampled by sentence-embedding similarity with sentiment alignment, making the task difficult enough to separate reasoning strategies.","core_discovery":"The central discovery is that the reasoning pattern an LLM uses during behavior simulation determines its accuracy, and that the effective pattern can be trained explicitly. On the FineRob benchmark, the authors identify 'role stereotype-based reasoning'—which over-relies on the persona profile—and 'observation and memory-based reasoning'—which relates candidate behaviors to the user's past actions. Advanced models such as GPT-4o naturally favor the latter, and similarity analysis of reasoning traces shows that higher alignment with role profiles does not improve accuracy, while alignment with history and options does. The paper's OM-CoT method operationalizes this by fine-tuning with chain-of-thought traces reorganized into <ANA> (observation analysis) and <MEM> (memory analysis) sections, yielding consistent F1 gains over standard CoT fine-tuning and outperforming few-shot prompting that already demonstrates the preferred pattern.","pith_inferences":["The <ANA>/<MEM> split is a generic inductive bias for personalization; the same two-token structure could be applied to recommendation or content generation tasks where the model must combine a current query with a user's history.","The oracle CoT generation makes OM-CoT dependent on a strong teacher model; as teachers improve or become cheaper, the method's ceiling likely rises, which is testable.","The decline beyond 30 history entries suggests a recency prior is being learned; an explicit recency-weighting mechanism might do even better than the fixed window.","Because the benchmark is multiple-choice with similarity-sampled distractors, gains may partly reflect discrimination between near-synonymous options rather than open-ended generation; extending FineRob to generative evaluation would clarify what the structured reasoning actually buys."],"forward_implications":["If OM-CoT's gains hold, behavior simulation should be treated as a reasoning task that can be improved through fine-tuning, not just through larger base models.","Because training and test users do not overlap, the learned observation-and-memory reasoning pattern transfers to users the model has never seen.","Ablations show role history is the dominant prompt component, so systems that compress or retrieve the right history will matter more than persona descriptions.","Performance peaks around 30 recent behaviors and declines with more, suggesting noise from long histories is a real cost in simulation.","The token ablation indicates sub-tasks need different balances, implying specialized prompting or routing by behavior element could extract further gains."],"supporting_citations":[{"why":"Closest prior work on persona-driven decision simulation; the macro-level benchmark that FineRob extends to fine-grained, real-world behavior.","marker":"[25]"},{"why":"Establishes the agenda of simulating human behavior with LLM-driven agents, which this paper applies to social-media users.","marker":"[18]"},{"why":"Prior evidence that LLMs can mimic real user preferences, providing the foundation for treating behavior simulation as a task.","marker":"[22]"},{"why":"Supplies the LoRA parameter-efficient fine-tuning method used in all OM-CoT experiments.","marker":"[11]"},{"why":"Mistral-7B is one of the two base models on which OM-CoT is evaluated against baselines.","marker":"[12]"}],"fun_headline_variants":["FineRob: 78.6k questions show LLMs simulate social media users better with memory","Observation beats stereotype: OM-CoT fine-tuning improves LLM social behavior simulation","New benchmark FineRob: LLM role-play improves via observation-memory reasoning","LLMs predict social media users better when reasoning from history, not stereotypes","OM-CoT: fine-tune LLMs to observe, then recall user history for accurate role simulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The FineRob benchmark's correct answers and its multiple-choice wrong answers are generated automatically, without human verification, so if those labels are noisy or the distractors are not genuinely plausible, the measured accuracy gains of OM-CoT might not reflect real behavior-simulation ability.","fun_headline_variants_meta":{"raw":{"variants":["FineRob: 78.6k questions show LLMs simulate social media users better with memory","Observation beats stereotype: OM-CoT fine-tuning improves LLM social behavior simulation","New benchmark FineRob: LLM role-play improves via observation-memory reasoning","LLMs predict social media users better when reasoning from history, not stereotypes","OM-CoT: fine-tune LLMs to observe, then recall user history for accurate role simulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3356,"prompt_tokens":906,"completion_tokens":2450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2341}},"tokens_in":522,"tokens_out":2450,"duration_ms":15746,"temperature":1.0,"reasoning_tokens":2341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:42:07.334743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of FineRob test items and have human raters decide whether the recorded behavior is what the user would most plausibly do; if agreement with the automatic labels is low, the benchmark scores are not measuring real behavior simulation. Alternatively, fine-tune with the CoT split at random positions into two labeled segments instead of the true <ANA>/<MEM> split; if accuracy does not drop, the semantic roles of the tokens are not the cause.","supporting_citations":[{"cited_title":"In: UIST","cited_arxiv_id":null,"evidence_quote":"Establishes the agenda of simulating human behavior with LLM-driven agents, which this paper applies to social-media users."},{"cited_title":"In: ICLR","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA parameter-efficient fine-tuning method used in all OM-CoT experiments."}],"review_version":1}