{"id":"35de0b94-3d2e-475b-950b-0c5a0ca9f682","arxiv_id":"2501.03276","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ComMer compresses and merges user documents into compact inputs that improve personalized skill learning under tight inference budgets, at the cost of detail on knowledge-intensive tasks.","lead":"This paper presents ComMer, a framework that compresses a user's documents into compact representations, merges them, and feeds them into a frozen language model for personalization. The authors report better quality on personalized writing tasks under tight inference budgets, while noting that knowledge-intensive tasks lose too much detail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core quality claim is budget-dependent and undefined: 'constrained inference budget scenarios' lacks a stated token or cost threshold, so ComMer's skill-learning win could be an artifact of comparing against artificially truncated baselines.","rationale":"The reader's verdict was UNVERDICTED because the full text, experimental details, baselines, and artifacts could not be checked. My stress-test agrees with that verdict: the central claim is not verifiable from the abstract. The reader's weakest assumption concerned whether the compressed representation preserves style and task cues while discarding factual details; that is a reasonable concern, but I see a more operationally load-bearing issue: the qualifier 'constrained inference budget scenarios' is undefined, and the claimed superiority cannot be assessed without knowing how the budget was set and whether baselines were matched on cost. If the full text specifies the budget and shows quality-budget curves, this concern may be resolved; if not, the claim is unfalsifiable. Because the reader already marked the paper UNVERDICTED, my concern does not change the verdict; it strengthens the reason for withholding acceptance until the evaluation protocol is disclosed.","tokens_in":685,"tokens_out":2595,"duration_ms":25029,"concrete_test":"On the LaMP tweet-paraphrasing and personalized news headline tasks, reproduce ComMer and a prompt-based baseline while sweeping the allowed input budget (for example, 256, 512, 1024, and 2048 tokens) and holding the output budget fixed. Report quality versus budget curves with error bars. If the prompt baseline matches or exceeds ComMer at any modestly larger budget, or if ComMer's advantage disappears when baselines are allowed the same total input tokens, then the 'superior in constrained inference budget scenarios' claim is a budget-selection artifact. Also report whether ComMer's merged representation is textual or vector-valued, and rerun the comparison with a text-summary baseline of equal token budget to isolate the merge effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Based on the abstract alone, the central claim is 'in constrained inference budget scenarios ComMer achieves superior quality in skill learning tasks.' For this claim to be meaningful, the evaluation must define the budget precisely (input tokens, output tokens, compute, or latency) and must compare ComMer against baselines under the same budget. The abstract does neither. If the prompt-based baseline is given a smaller effective context or a token cap that forces it to drop user information, while ComMer spends its entire budget on a compressed representation, the comparison conflates compression quality with budget allocation. The claim would then be an artifact of selecting a favorable budget point rather than a property of the compression and merging method. A second, related gap is that 'merged and fed into a frozen LLM' does not state whether the merged representation is textual (for example, a generated summary) or a vector or embedding; if it is an embedding, comparison with text prompts introduces a modality confound, and if it is textual, the compression step is essentially summarization and the skill-learning gains may depend on summarization quality rather than the merge mechanism. Without budget curves and modality specification, the strongest claim is unfalsifiable from the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ComMer, a framework that compresses users' documents into compact representations, merges the representations, and feeds them into a frozen LLM for personalization. The abstract reports evaluations on two task types: personalized skill learning (tweet paraphrasing and news headline generation from the LaMP benchmark) and knowledge-intensive question answering (PerLTQA). It claims that ComMer achieves superior quality in skill learning under constrained inference budgets, while acknowledging limitations in knowledge-intensive settings due to loss of detailed information. The reviewable text consists only of the abstract; no methodological or experimental details are available.","tokens_in":905,"tokens_out":2215,"duration_ms":22246,"significance":"If the central claim is substantiated, ComMer would offer a meaningful efficiency/quality trade-off for personalizing frozen LLMs, reducing inference cost relative to prompt-based methods and avoiding fine-tuning overhead. The paper explicitly identifies a limitation in knowledge-intensive tasks, which is a useful honest boundary. However, the abstract alone cannot establish significance: it reports no effect sizes, error bars, baseline comparisons, significance tests, or precise budget definitions. The claimed advantage is conditional on an undefined 'constrained inference budget scenario,' and the representation modality is unspecified. The work is potentially valuable but currently not assessable from the provided text.","major_comments":[{"comment":"The central claim that 'in constrained inference budget scenarios ComMer achieves superior quality' is unfalsifiable without a precise definition of the inference budget (e.g., number of tokens, compute, or latency). The abstract does not state the budget threshold or provide budget curves, so the reported advantage could be an artifact of comparing against artificially truncated baselines rather than a property of the compression and merging method.","section":"Abstract"},{"comment":"The sentence 'compressing users' documents into compact representations, which are then merged and fed into a frozen LLM' does not specify whether the merged representation is textual (e.g., a generated summary) or a vector/embedding. If it is an embedding, comparing against text prompts introduces a modality confound; if it is textual, the method reduces to summarization and the specific contribution of the merge step is unclear. This ambiguity must be resolved before the reported results can be interpreted.","section":"Abstract"},{"comment":"The evaluation section names three datasets but reports no quantitative outcomes: no effect sizes, error bars, or statistical significance for the claimed 'superior quality' in skill learning, nor for the 'limitations' in knowledge-intensive settings. Without any reported numbers or baseline comparisons, the abstract's central conclusions are unsupported in the reviewable text.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract says 'two types of personalization tasks' but lists three datasets; clarify that there are two task categories (skill learning and knowledge-intensive) and three datasets.","section":"Abstract"},{"comment":"The phrase 'resource and computational constraints' is vague; specify which resources are meant (context window, inference cost, update cost) so the motivation is concrete.","section":"Abstract"},{"comment":"Consider reporting at least one quantitative result and the exact budget configuration in the abstract, so that the main claim is testable from the abstract alone.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"The manuscript text provided to the referee contains only the abstract; the full text is absent in the reviewable version. This makes a complete soundness assessment impossible. The abstract's central claim depends on unspecified experimental conditions (budget definition, representation modality, baselines). I cannot recommend acceptance based on the abstract alone. If the full manuscript is available, it should be evaluated for the specifics of the budget curves, baseline fairness, and whether the reported results include significance tests and error bars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: I only have the abstract, so this is a review of a one-paragraph claim, not a paper. The idea is coherent and worth taking seriously if the full text defines the budget and the baselines carefully. The compression-plus-merging framing is a real departure from the two defaults most personalization work uses, prompt stuffing and fine-tuning. Keeping the LLM frozen and representing a user as a compact merged artifact attacks a legitimate cost bottleneck. Credit is also due for reporting that the method loses on knowledge-intensive tasks; not every abstract admits its own limitation.\n\nSoft spots: the central claim is budget-dependent and the abstract never says what the budget is. 'Constrained inference budget scenarios' could mean anything. To be meaningful, the evaluation needs budget curves, a definition of budget (input tokens, total tokens, cost, or latency), and baselines run under the same budget. If the prompt baseline is starved of context while ComMer gets a compressed representation of the whole user history, the comparison conflates compression with budget allocation. The abstract also doesn't say whether the merged representation is text or a vector. If it's an embedding, you have a modality confound against text prompts; if it's text, the compression step is close to summarization and the skill gains may ride on summarization quality rather than on the merging mechanism. No effect sizes, baselines, or error bars are visible, so the strength of the quality claim can't be judged.\n\nThese are abstract-level gaps, not demonstrated defects. The advertised method could be solid; I just can't tell from what's in front of me. The citation pattern is clean, there's no visible circularity, and the external benchmarks are appropriate.\n\nWho it's for: people working on efficient personalization for LLMs, especially cost-sensitive deployment. If the full paper supplies budget curves, baseline parity, and a clear description of the representation, it deserves referee time and could be a useful addition. I would not cite it from the abstract alone, but I'd read the full version. Send it to peer review with the expectation that the authors tighten the budget definition and the comparison.","headline":"Abstract-only look at a coherent compression-and-merge personalization idea whose main claim is untestable until the budget and baselines are pinned down.","tokens_in":1384,"tokens_out":1855,"would_cite":false,"duration_ms":18218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ComMer claims that compressing each user's documents into compact representations, merging them, and feeding the result to a frozen LLM delivers better quality on personalized skill-learning tasks than prompt-based or fine-tuned baselines…","keywords":["personalization","large language models","prompt compression","document merging","frozen model","skill learning","knowledge-intensive tasks","inference budget"],"falsifier":"Take a single user whose target style appears only in one tweet in the training set, compress all of that user's documents together, and ask the frozen model to paraphrase a new tweet in that style; if the compressed representation loses the only style exemplar and quality collapses, the claim that compression preserves skill cues would be falsified.","tokens_in":530,"feed_emoji":"🧊","tokens_out":2813,"duration_ms":26652,"temperature":0.7,"pith_summary":"This paper introduces ComMer, a framework that personalizes a frozen large language model by compressing each user's documents into short representations, merging those representations across users, and feeding the merged text into the model. The central claim is that in settings where inference budget is constrained, this compress-and-merge procedure produces better results on personalized skill-learning tasks such as tweet paraphrasing and news headline generation than exposing raw documents through the prompt or fine-tuning the model. The same experiments show that the approach falls short on knowledge-intensive tasks because compression discards too much detailed information. If the claim holds, personalization can be made cheaper at inference time without per-user fine-tuning, at the cost of factual recall.","feed_headline":"Compress-and-merge personalization beats prompts for skill tasks","feed_subtitle":"A frozen LLM fed compact merged user summaries wins on style tasks under tight inference budgets, at the cost of factual recall.","key_machinery":"The central object is the compress-then-merge representation: each user's documents are compressed into a short text that captures style and task-relevant cues, and those short texts are concatenated into one merged prompt passed to a frozen LLM. The compression step is what keeps the model frozen and inference cost low, while the merging step pools information from multiple users in a single forward pass. This replaces both the long raw-document prompt and any per-user weight update.","core_discovery":"ComMer's central claim is that a compact, merged representation of user documents is sufficient for a frozen LLM to learn the style and task behavior of that user. When the inference budget is tight, this compressed representation outperforms both prompt-based personalization that spells out the full documents and fine-tuned models on the evaluated skill-learning tasks. The mechanism works by compressing each user's documents separately, then merging the compressed texts into a single prompt-like input. The trade-off is explicit: on knowledge-intensive question answering, the compressed representation loses too much detail and underperforms, revealing the boundary of the method.","pith_inferences":["The compress-then-merge design suggests a general principle: for tasks that hinge on style, register, or formatting, a lossy summary can be more efficient than exhaustive context, whereas tasks that hinge on facts need a fidelity-preserving path; this could extend to code style, email tone, or other personalized formatting tasks.","The merging step implies that individual user representations could be cached and combined at serving time, acting as a lightweight personalization cache; the paper does not explore this operational reading.","A testable extension would decouple the compression ratio per user: reserving more tokens for users whose style is sparse or unusual could recover some skill-learning quality, an optimization hinted at but not pursued in the paper."],"forward_implications":["In constrained inference budgets, ComMer yields better quality than prompt-based or fine-tuned baselines on personalized skill-learning tasks, enabling lower per-request computation for style-oriented personalization.","The method's value is task-dependent: it works for style-transfer-like skills such as tweet paraphrasing and headline generation but degrades on knowledge-intensive tasks, so deployment should match the task type.","Because the LLM stays frozen, keeping personalization current only requires re-compressing and re-merging documents, avoiding expensive re-training when user data changes."],"supporting_citations":[],"fun_headline_variants":["Compress-merge beats prompts for LLM personalization under tight budgets","Frozen LLM learns user style from compressed merged docs","ComMer: Skill gains from compressed personalization, factual recall suffers","Efficient personalization: merge compact user summaries for frozen LLMs","Compressed user data boosts style tasks, not fact-heavy ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The compressed and merged representation of a user's documents keeps the stylistic and task-specific cues that personalization needs, even though it discards most of the original text.","fun_headline_variants_meta":{"raw":{"variants":["Compress-merge beats prompts for LLM personalization under tight budgets","Frozen LLM learns user style from compressed merged docs","ComMer: Skill gains from compressed personalization, factual recall suffers","Efficient personalization: merge compact user summaries for frozen LLMs","Compressed user data boosts style tasks, not fact-heavy ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1897,"prompt_tokens":863,"completion_tokens":1034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":945}},"tokens_in":479,"tokens_out":1034,"duration_ms":8639,"temperature":1.0,"reasoning_tokens":945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:10:24.080639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single user whose target style appears only in one tweet in the training set, compress all of that user's documents together, and ask the frozen model to paraphrase a new tweet in that style; if the compressed representation loses the only style exemplar and quality collapses, the claim that compression preserves skill cues would be falsified.","supporting_citations":[],"review_version":1}