{"id":"34c267e1-fa94-4f25-8e10-f4fe99d1b0ff","arxiv_id":"2608.04095","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new event-grounded benchmark shows LLM agents remain far below saturation on personalized financial memory, with simple retrieval outperforming structured memory systems especially after consequential events.","lead":"FinPerMA is a new benchmark that tests whether LLM-based financial advisors can track and update a user's preferences across events, using 2,994 questions from 276 synthetic investor personas. The authors report that even the strongest models stay below about 47% accuracy, and that simple retrieval often beats purpose-built memory systems, especially after a shock.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold labels rest on unvalidated Impact Model rules and LLM narration; without human validation of answerability, the ~0.47 ceiling and Post-Shock gap may measure label noise, not memory failure.","rationale":"The paper is internally coherent: the corpus is frozen, MCQ grading is deterministic, the generation pipeline is auditable, and the authors honestly enumerate limitations such as one-seed evaluation and unmatched token budgets. Those limitations are real, but the decisive one is the validity of the gold labels, because every headline number is a comparison against them. The reader's weakest assumption names exactly this concern, and my analysis agrees while sharpening it to the Post-Shock condition: since the checkpoint question is posed immediately after the event, it can be answered from local context, so the checkpoint must be shown to require integration across sessions rather than recency. The proposed human-annotation study on the frozen corpus is a single check that would settle whether the 47% ceiling and the widening gap reflect genuine model incapability or label artifacts. Because this concern is already reflected in the reader's conditional verdict, no further verdict change is needed.","tokens_in":10290,"tokens_out":5347,"duration_ms":59267,"concrete_test":"Draw a stratified sample of about 100 Post-Shock and 100 in-timeline MCQs from the frozen v8gold set. Give 3–5 financially literate human annotators the same full_context inputs (persona, timeline, dialogues, question, options) with the gold answer withheld, and ask them to pick the option best matching the persona's updated preferences. Compare human accuracy to the best full-context LLM (about 47%). In a second arm, give annotators identical inputs minus the post-event dialogue, to verify the event is necessary and answers are not leaked. If human accuracy on full inputs is substantially above 47% and drops to near chance without the event dialogue, the labels are valid and the central claim stands. If human accuracy is near the LLM ceiling or does not drop in the no-event arm, the benchmark labels are unreliable and the Post-Shock comparison should be re-interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claims—no full-context configuration exceeds approximately 0.47 overall or 39% on MCQ, and the full-context vs mem0 gap widens from 8.0 in-timeline to 13.0 at Post-Shock—are only interpretable if gold labels are correct and the Post-Shock questions are answerable from the provided evidence. The labels are produced by hand-set Impact Model rules (Eqs. 3–6: event-type coefficients, Big-Five multiplier 0.30/−0.15, lambda=1.3, threshold 0.3) that are explicitly described as not an empirically estimated model of human behavior, then realized by an LLM narrator and screened by automated checks. No human validation is reported: there is no check that the dialogues actually exhibit the intended preference change to a human reader, that the reactions are inferable from the dialogue without relying on the hidden rule, or that the questions are not answerable from local post-event text alone. If the deterministic rules produce reactions that no competent advisor would infer, or if narration leaks answers via wording, then a 47% ceiling and a widening Post-Shock gap would reflect artifacts of the generator and question design rather than a genuine failure to maintain and update a user model. The Limitations section concedes this risk, but the paper's central conclusion is stated without that caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinPerMA, a benchmark for evaluating LLM agents' personalized memory over longitudinal investor trajectories. The generation pipeline combines deterministic, theory-informed Impact Model rules (Eqs. 3–6), controlled LLM narration, and automated quality screening to produce a frozen corpus of 2,994 questions from 276 personas, including a Post-Shock checkpoint intended to test whether an agent integrates a material event into its persistent user model. Evaluation of seven frontier LLMs and up to seven memory configurations shows that no full-context configuration exceeds roughly 0.47 overall accuracy or 39% on multiple-choice questions, that retrieval-based memory approaches the full-context upper bound more efficiently than structured profile systems, and that the gap between full-context and mem0 widens from 8.0 in-timeline to 13.0 at the Post-Shock checkpoint. The paper concludes that memory systems often fail to update a user model after consequential events.","tokens_in":10571,"tokens_out":4588,"duration_ms":48531,"significance":"If the benchmark's gold labels are valid, FinPerMA addresses a real gap in personalized-memory evaluation by focusing on event-conditioned preference updating, and the Post-Shock checkpoint is a useful protocol contribution. The frozen corpus, deterministic rules, and release of code and prompts are concrete strengths that make the resource reproducible and amenable to future comparison. The empirical finding that all evaluated models remain far from saturated is striking and, if grounded in reliable labels, would be a valuable reference for memory-system design. However, the lack of human validation of the generated trajectories and labels, the single evaluation seed, and the unmatched token budgets across memory configurations temper the strength of the central conclusions; these issues are load-bearing rather than cosmetic.","major_comments":[{"comment":"The gold labels are produced by hand-set Impact Model rules that the paper explicitly states are not an empirically estimated model of human behavior, and no human validation of the resulting trajectories, question answerability, or label correctness is reported. The abstract and §3.4 conclude that memory systems 'fail to integrate new evidence into a persistent user model,' but if the deterministic rules or LLM narration inject noise or if questions are not answerable from the provided evidence, then the observed ~0.47 ceiling and the Post-Shock gap would be artifacts of the generator rather than measurements of memory capability. The authors should report a human spot-check on a sample of questions (e.g., 100–200), including agreement on whether the intended preference change is inferable from the dialogue and whether the gold answer is unambiguously correct, and show that the main findings are unchanged on the human-validated subset.","section":"§2.4 (Eqs. 3–6) and Limitations"},{"comment":"The claim that the full_context−mem0 gap widens from 8.0 in-timeline to 13.0 at Post-Shock compares configurations with unmatched native token budgets: full_context uses 12.8k context tokens per query while mem0 uses 1.02k. The widening gap may simply reflect the smaller evidence budget available to mem0 at the Post-Shock checkpoint rather than a specific failure of memory updating. A token-controlled comparison, such as supplying the same number of retrieved tokens to each configuration or ablating the evidence budget within full_context, is needed to support the interpretation that the Post-Shock checkpoint isolates memory-updating failure as opposed to general evidence insufficiency.","section":"§3.3 and Table 1, Panel B"},{"comment":"The memory-system comparison uses a single evaluation seed per configuration. Without repeated runs, confidence intervals, or significance tests, the reported differences among memory systems (e.g., Memobase vs Mem0, or the 8.0→13.0 gap) could be within sampling noise. Because the benchmark is frozen at 2,994 questions, one could bootstrap the question set or run multiple evaluation seeds with different random orders; either would provide variance estimates. The authors should report at least three seeds or bootstrap confidence intervals for the key comparisons in Table 1 and §3.3.","section":"§3.1 and Limitations"},{"comment":"Open-ended correctness is determined by majority vote of a three-way cross-vendor judge ensemble, with no validation of judge accuracy against human labels. Since 500 of the 2,994 questions are open-ended and the open-ended accuracy is a major driver of the memory-system ranking (retrieval's advantage on Open, and the MemFid scores used in attribution), judge bias could propagate to the central conclusions. The authors should report judge–human agreement on a human-annotated sample (e.g., Cohen's kappa or accuracy), or at least a secondary human evaluation on a subset of open-ended answers, to establish that the judge scores are reliable.","section":"§3.1 (Metrics) and Table 1"}],"minor_comments":[{"comment":"Model names are inconsistent between the text and table (e.g., 'Qwen-3.8' in the table caption vs 'Qwen3.8' in the text, and 'Open' is used as a column header without a definition in the metrics paragraph). Please standardize naming and spell out each column abbreviation.","section":"§3.1 and Table 1"},{"comment":"The parenthesis '(novel)' in 'Post-Shock (novel)' is not explained; clarify whether 'novel' means the question is not seen during the dialogue or that the scenario is new.","section":"§2.6"},{"comment":"The abstract reports 'approximately 0.47 overall accuracy' while Table 1 gives a maximum of 0.469; consider rounding consistently or reporting exact values in the abstract.","section":"Abstract and Table 1"},{"comment":"The abbreviations 'PAS' and 'BIA' are used without expansion; define them at first use, perhaps as 'preference alignment score' and 'bias identification accuracy' if that is intended.","section":"§3.1"},{"comment":"The radar and distractor-trap subfigures have small labels; consider increasing font sizes and adding a legend clearly mapping line styles to model names for readability.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an AI/ML venue and the resource could be useful to the community. The main concerns are about the validity of the gold labels and the statistical support for the headline claims. The authors should be encouraged to add a human validation study on a sample of questions, run multiple evaluation seeds, and include token-controlled comparisons for the Post-Shock analysis. These are addressable within the manuscript's scope, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FinPerMA is worth your time. The new thing is the Post-Shock checkpoint: a clean way to ask whether an agent has integrated a consequential event into its persistent user model, not just recalled facts. That is a genuinely useful addition to the LLM-memory benchmark landscape, and the paper does it carefully. The three-layer Impact Model is transparent, the corpus is frozen, and they ship the rule engine, seeds, and prompts. That reproducibility is real evidence and should be credited.\n\nThe evaluation is also solid in design: seven backbones, multiple memory architectures, a no-memory lower bound, and a full-context upper bound. The attribution analysis separating factual recall from preference reasoning is informative, and the finding that retrieval can beat purpose-built memory systems after shocks is worth discussing.\n\nNow the soft spots, in proportion. The central validity concern is exactly what the stress-test flags: the gold labels come from hand-set rules (Eqs. 3–6) plus LLM narration, with no reported human validation. If the narrator leaks answers or the deterministic rules produce reactions no competent advisor would infer, then the ~0.47 ceiling and the widening Post-Shock gap measure artifacts of the generator, not memory failure. The authors acknowledge this in the Limitations section, but the abstract and conclusion state the results without that caveat. That is an honest but important gap.\n\nThe other weaknesses are minor and mostly acknowledged: one seed per memory configuration, unmatched native token budgets, and LLM-judge grading of open-ended answers. None of these are fatal on their own, but together they mean the quantitative claims should be treated as provisional.\n\nAll that said, the paper is clearly a serious piece of work. The authors are transparent about what is estimated and what is not, and they release enough artifacts for others to probe the validity questions directly. This deserves a real referee, and a good referee would ask for a human spot-check of answerability and label consistency, a sensitivity analysis on the rule coefficients, and multi-seed runs. If those come through, the benchmark could become a standard tool. As it stands, I would cite it and bring it to a reading group.","headline":"FinPerMA's Post-Shock checkpoint is a real contribution to LLM memory evaluation, but its headline numbers should be read as provisional until the synthetic gold labels get a human spot-check.","tokens_in":11081,"tokens_out":1314,"would_cite":true,"duration_ms":16617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinPerMA: a benchmark that tests whether LLM agents update a user model after financial events, and shows they largely do not.","keywords":["personalized memory benchmark","LLM agents","event-driven preference adaptation","Post-Shock checkpoint","Impact Model","financial advisory","longitudinal investor personas","memory systems evaluation"],"falsifier":"Take the frozen FinPerMA corpus and re-answer the Post-Shock questions with the true post-shock preference state (the Impact Model's intended updated values) injected directly into each memory system; if accuracy does not rise well above the reported ~39% MCQ, the benchmark is testing inference or recall rather than memory integration. A complementary check is to replace the rule-based gold labels with human-annotated or real-investor preference changes and see whether the relative ranking of retrieval versus structured memory reverses.","tokens_in":10078,"feed_emoji":"🧠","tokens_out":8338,"duration_ms":79041,"temperature":0.7,"pith_summary":"This paper argues that current LLM agents are not actually maintaining a persistent, updated model of a user, and builds a benchmark to measure that claim. FinPerMA creates 276 synthetic investor personas with dated event timelines, generates dialogues with deterministic rules that specify how preferences should change after financial shocks, and freezes the corpus so every system faces the same trajectories. On 2,994 questions, no system—even one given the entire conversation history as context—exceeds about 47% overall accuracy or 39% on multiple-choice questions. The widest gaps appear right after a material event, when a memory system should revise a user's risk profile. The paper concludes that current memory architectures often preserve facts while losing the preference signals that personalization requires.","feed_headline":"No LLM agent tops 47% on event-updated investor memory","feed_subtitle":"A new 2,994-question benchmark shows why assistants fail to revise user preferences after shocks: summaries drop the evidence.","key_machinery":"The load-bearing object is the three-layer Impact Model. Layer 1 deterministically builds an ImpactConstraint for each persona-event pair: a four-way Behavioral Investor Type gates direction and forbidden biases, an event-type coefficient $c[\\text{type}] \\cdot \\text{severity}/5$ sets magnitude, a Big-Five multiplier $1 + 0.30(P_N - 0.5) - 0.15(P_E - 0.5)$ scales it, an experience-history aggregation with weight exponent $\\lambda = 1.3$ modulates it, and a symbolic memory-reference anchor (threshold $0.3$ on tag-Jaccard plus event-type match) supplies a reference value. Layer 2 narrates an event reaction under these hard constraints, and Layer 3 validates with automated checks and up to three retries. The Post-Shock checkpoint, posed immediately after a high-severity event, is what separates recall of the past from integration of new evidence into a persistent user model.","core_discovery":"The paper's central claim is that event-conditioned preference updating is a distinct capability that current LLM agents lack. On a frozen corpus of 2,994 questions from 276 personas, giving a model the entire conversation history as context lifts accuracy to only about 0.47 overall and about 39% on multiple choice, and the gap between full-context models and structured-memory systems grows from 8.0 points in-timeline to 13.0 points at the Post-Shock checkpoint. This is evidence that memory systems fail to integrate a material event into a persistent user model. The paper also claims that summary/profile memory preserves factual details at the expense of preference signals, so simple retrieval of raw dialogue recovers roughly 88% of the no-memory-to-full-context gap with about a tenth of the context tokens.","pith_inferences":["If the central claim holds, the Post-Shock protocol could be adapted as a general stress test for any long-horizon assistant (medical, legal, educational) whose advice should change after a material event.","A testable design implied by the paper: separate stable attributes from mutable preferences, timestamp preference deltas, and retrieve raw dialogue for preference-sensitive queries; this should close part of the gap without full context.","The deterministic Impact Model could double as a controllable data generator for counterfactual training, since it can produce trajectories with known preference updates on demand.","The absence of human validation of gold labels is the main external-validity threat; comparing the generated trajectories to real investor-panel or human-annotated preference changes would settle whether the Post-Shock results measure memory updating or generator artifacts."],"forward_implications":["Full context alone does not solve personalization: even the strongest configuration leaves a wide accuracy gap, so the limiting factor is integration and reasoning, not context-window size.","Lightweight retrieval of verbatim dialogue recovers most of the benefit of full context at a fraction of the tokens, so memory systems should not summarise away the evidence that preference reasoning needs.","Structured/profile memory systems that consolidate facts are likely to keep failing on tasks that require inference or updating, because they discard preference signals.","The Post-Shock gap being wider than the in-timeline gap means memory systems have a specific, testable weakness in integrating consequential events into the user model.","The anti-typical-persona gap indicates that agents default to type-level stereotypes when individual evidence conflicts with a familiar profile, so personalization needs an override mechanism."],"supporting_citations":[{"why":"Supplies the Behavioral Investor Type classification that gates the direction and forbidden biases in the Impact Model's Layer 1.","marker":"Pompian 2012"},{"why":"Motivates the experience-history aggregation and age-dependent weighting used to modulate event impact.","marker":"Malmendier and Nagel 2011, 2016"},{"why":"Motivates the symbolic memory-reference anchor and reference-dependent value aggregation in Eq. (6).","marker":"Bordalo, Gennaioli, and Shleifer 2020"},{"why":"Provides the time-varying risk-response evidence behind the event-shock direction and magnitude coefficients.","marker":"Guiso, Sapienza, and Zingales 2018; Cohn et al. 2015"},{"why":"Supplies the Big-Five personality associations used in the trait multiplier.","marker":"Durand, Newby, and Sanghani 2008"},{"why":"Frames both the retrieval-augmented baselines and the constraint-augmented narrator prompt.","marker":"Lewis et al. 2020"},{"why":"Provides the LLM-as-judge method used to grade open-ended answers and quality scores.","marker":"Zheng et al. 2023"},{"why":"Defines the prior personalized-memory benchmark whose anti-typical gap this paper reproduces and extends.","marker":"Jiang et al. 2025"},{"why":"Supplies LongMemEval, the fact-level temporal/update benchmark that FinPerMA contrasts with on event-conditioned updating.","marker":"Wu et al. 2025"}],"fun_headline_variants":["LLM agents cap at 47% on event-updated investor memory","Investor-memory benchmark: summaries lose preference signals after shocks","Full context beats memory systems by 13 points post-shock in FinPerMA","Retrieval recovers 88% of memory gap with 10% of tokens","Post-shock investor memory: simple retrieval outperforms summary memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's gold labels stand or fall on the assumption that the hand-written Impact Model rules—event-type coefficients, Big-Five multiplier weights 0.30 and -0.15, experience exponent $\\lambda = 1.3$, and memory threshold 0.3—are a valid stand-in for how real investors change preferences after events, so the generated dialogues and labels track true updating rather than generator artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents cap at 47% on event-updated investor memory","Investor-memory benchmark: summaries lose preference signals after shocks","Full context beats memory systems by 13 points post-shock in FinPerMA","Retrieval recovers 88% of memory gap with 10% of tokens","Post-shock investor memory: simple retrieval outperforms summary memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2807,"prompt_tokens":913,"completion_tokens":1894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":529,"tokens_out":1894,"duration_ms":15102,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:33:28.684755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the frozen FinPerMA corpus and re-answer the Post-Shock questions with the true post-shock preference state (the Impact Model's intended updated values) injected directly into each memory system; if accuracy does not rise well above the reported ~39% MCQ, the benchmark is testing inference or recall rather than memory integration. A complementary check is to replace the rule-based gold labels with human-annotated or real-investor preference changes and see whether the relative ranking of retrieval versus structured memory reverses.","supporting_citations":[{"cited_title":", title =","cited_arxiv_id":null,"evidence_quote":"Supplies the Behavioral Investor Type classification that gates the direction and forbidden biases in the Impact Model's Layer 1."}],"review_version":1}