{"id":"c15dc10c-4998-479d-881b-174017c2f538","arxiv_id":"2508.08634","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Explicit per-turn personalization level detection plus per-level weighted fusion of personalized and non-personalized query rewrites improves retrieval on TREC iKAT 2023 and 2024.","lead":"A conversational search system that decides, turn by turn, whether to personalize results using a user profile, and then blends personalized and general query rewrites with different weights, improves retrieval on two TREC benchmarks. The method asks an LLM to label each query's personalization need, then tunes fusion weights per label on one dataset and applies them to the other.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablation: per-level fusion weights are never compared to a single globally optimized weight triple, so the claimed adaptive mechanism may not be what drives the gains.","rationale":"The reader's weakest assumption was cross-dataset transfer of per-level weights. I agree that transfer is untested in a statistically rigorous way, but I see an even more basic missing control: there is no comparison between per-level weights and a single globally optimized weight triple. The claimed mechanism is that the fusion weights depend on the predicted personalization level. The current ablations compare against weight schemes that either ignore validation data entirely (Random, Equal) or use different heuristics (Entropy, DEPS), and against fusion strategies that do not optimize weights at all (RRF, round-robin, representation fusion). None of these rules out the simple explanation that any validation-based weight selection improves results, independent of level conditioning. The reported per-level triples are close, which makes that alternative explanation concrete rather than speculative. The paper deserves credit for reporting aligned comparisons with fixed backbones and for using a cross-dataset weight transfer protocol, both of which strengthen the overall empirical claim. The concern is specifically about attributing the gain to adaptive personalization. This is a single, easily run ablation, so the appropriate verdict is conditional acceptance pending that check.","tokens_in":20325,"tokens_out":7030,"duration_ms":77882,"concrete_test":"Re-run Algorithm 1 with a single global level, i.e., collapse all turns into one group and select one weight triple on the validation collection using the same grid search and the same cross-dataset protocol: select on iKAT-23 and test on iKAT-24, then select on iKAT-24 and test on iKAT-23. Report MRR, NDCG@3, and Recall@100 alongside the per-level results in Table 1 and Table 2. If the global triple comes within about 0.5 NDCG@3 or MRR of the per-level triples on both collections, the personalization-level conditioning is not what drives the reported improvement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central contribution is adaptive personalization: fusion weights in Eq. (5) are conditioned on the LLM-predicted personalization level. But the experiments never isolate this conditioning. Algorithm 1 optimizes three separate weight triples, one per level, while Table 2 compares against Random, Equal, User-based Entropy, and DEPS, and Table 4 compares against reciprocal rank, round-robin, and representation fusion. None of these baselines is allowed to optimize a single global weight triple on the same validation data with the same grid search. The reported triples are numerically close within each dataset, e.g., iKAT-24 gives (0.20, 0.38, 0.42), (0.28, 0.36, 0.36), and (0.23, 0.40, 0.37). It is therefore plausible that one globally optimized triple captures almost all of the gain, and that the per-level conditioning contributes little or nothing. If that is true, the improvement over Equal Personalization comes from tuning weights on a labeled collection, not from adapting to the personalization level, and the paper's core 'adaptive personalization' explanation of the Table 1 gains is not established. The weak LLM-human agreement in Table 5 makes this concern more acute: if level labels are noisy, the three fitted triples may just be three noisy estimates of the same underlying optimal weights. A secondary issue is that the metric M in Eq. (5) is never specified; if weights are selected to maximize NDCG@3, then gains in MRR or Recall are not the optimized objective. The reader's cross-dataset transfer concern is real, but the missing global-weight control is more fundamental: before asking whether per-level weights transfer, one must show they differ from a single weight vector.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the question of when and how much to personalize conversational information retrieval. The authors propose APCIR, which first uses GPT-4o with chain-of-thought and in-context examples to label each query turn as non-personalized, partially personalized, or personalized; then generates up to three reformulated query variants per turn (non-personalized query, query plus pseudo-response, and personalized query plus pseudo-response, with a second non-personalized variant for non-personalized turns); and finally fuses the ranking lists with a linear combination whose weights are selected per personalization level by grid search on a validation collection and transferred to the other iKAT collection. Experiments on TREC iKAT 2023 and 2024 report large gains over existing conversational query reformulation baselines in both open and aligned comparisons, with additional analyses of ablation components, weight-identification baselines, fusion strategies, retrievers/re-rankers, and LLM-versus-human personalization judgments.","tokens_in":20675,"tokens_out":8975,"duration_ms":92463,"significance":"The paper makes a clear and practically relevant proposal: instead of always injecting a full user profile into query reformulation, it conditions reformulation and fusion on an explicit personalization level. The aligned comparisons with fixed query rewriter, retriever, and reranker are a strength, as is the cross-dataset protocol for selecting fusion weights without test-label leakage and the release of code and data. The reported improvements are large (e.g., MRR 57.3 vs. 45.7 on iKAT-23 and 82.7 vs. 43.2 on iKAT-24 in the open comparison), and I found no circularity in the described evaluation protocol: the LLM personalization labels are not derived from the relevance labels, and the weight-fitting collection is different from the test collection. The main reason the significance is not yet fully established is that the per-level adaptive mechanism is not compared with a single globally optimized fusion weight, and the optimization metric used for weight selection is unspecified; both are needed to attribute the gains to adaptation rather than to weight tuning.","major_comments":[{"comment":"The pseudocode and the experimental protocol are inconsistent. Algorithm 1 takes a single conversation session as input, builds level-specific lists L_l from that session, selects optimal weights by maximizing M on phi_fused(q_n) for the same turns in L_l (lines 14-28), and then applies these weights to the same turns (lines 29-32). If run as written, this fits and evaluates on the same query turns. The text in Sec. 5.3 instead says that weights are selected on one iKAT collection and applied to the other. Please rewrite Algorithm 1 to take a validation session or collection and a separate test session as inputs, and state clearly in the pseudocode where weight selection stops and test evaluation begins. This is load-bearing for every reported number.","section":"Algorithm 1 and Sec. 5.3"},{"comment":"The claimed adaptive mechanism is not isolated. The per-level weight triples are compared with Random, Equal, User-based Entropy, and DEPS, but none of these baselines is allowed to optimize a single global weight triple on the same validation queries using the same grid search. The three fitted triples are numerically close (e.g., iKAT-24: (0.20,0.38,0.42), (0.28,0.36,0.36), and (0.23,0.40,0.37)), so a single triple may account for most of the gain over Equal Personalization. Please add a global-weight baseline and an ablation that removes the level conditioning; without this, the paper's central claim that adaptively conditioning fusion weights on the personalization level drives the Table 1 gains is not established.","section":"Sec. 4.3, Eq. (5), Table 2"},{"comment":"The optimization metric M is never specified. It appears only as 'a retrieval evaluation metric' in Eq. (5) and as the accumulator in Algorithm 1. If M is, say, NDCG@3, then the MRR and Recall numbers in Tables 1-4 are not the objective being optimized, and the reported gains on those metrics could be incidental. Specify M, and ideally repeat the weight search for each reported metric to show that the conclusions are not metric-dependent.","section":"Sec. 4.3, Eq. (5), Algorithm 1"},{"comment":"No variance or repeated-run information is reported, and significance testing is incomplete. GPT-4o generation is stochastic, yet the paper reports point estimates and marks the dagger symbol only for selected comparisons in Table 1, with no significance tests for Table 2. Report means and standard deviations (or bootstrap confidence intervals) over multiple runs, state the sampling temperature used for the LLM calls, and apply significance tests to the weight-identification comparisons.","section":"Tables 1 and 2"},{"comment":"The cross-dataset transfer of per-level weights rests on the assumption that LLM-predicted levels are consistent enough across collections. The agreement rates in Table 5 are 77.8%/47.8% (iKAT-23) and 79.6%/63.0% (iKAT-24) for personalized/non-personalized classes; the paper attributes the gap to annotation discrepancy, but this is an interpretation, not a measurement. Please add a sensitivity analysis: fit weights on one split of each dataset and evaluate on the held-out split, and compare the results when the level assignment uses LLM labels versus human labels. This would separate level-prediction noise from the fusion benefit.","section":"Sec. 5.3, Table 5"}],"minor_comments":[{"comment":"The preliminary experiment lacks error bars, numeric values, and significance tests; please add them or state explicitly that the figure is illustrative.","section":"Sec. 3.2, Figure 2"},{"comment":"The reported fusion weight triples are not labeled with their personalization levels; please clarify the order (w1,w2,w3) and which level each triple corresponds to.","section":"Sec. 5.3"},{"comment":"The notation 'if l_n != a' and '{or q'_n,r'_n, q''_n,r''_n} if l_n == a' is ambiguous; rewrite these as explicit conditional statements.","section":"Algorithm 1, lines 5-6"},{"comment":"The summation 'sum_{n=1}^N ..., l_n = l' is ambiguous; clarify that the sum is over turns in level l, as in Algorithm 1.","section":"Eq. (5)"},{"comment":"The open comparison for iKAT-24 has '-' for PCIR and GtR; state explicitly that these numbers were not available and were therefore omitted, and consider reporting them with the same backbone if possible.","section":"Table 1"},{"comment":"The phrase 'The results confirm the effectiveness of adaptive personalization' overstates the evidence given the missing global-weight ablation; consider softening to 'suggest' or 'indicate' pending the new analysis.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"I do not see evidence of intentional circularity in the described protocol, but the Algorithm 1 pseudocode must be corrected to clearly separate validation and test turns. The main risk is that the per-level fusion weights contribute little beyond a single globally optimized triple; the requested ablation is within the manuscript's scope and should be feasible. The paper is otherwise within the scope of CIKM and the strengths (aligned comparison, cross-dataset transfer, code release) make it a worthwhile revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a real gap. The new thing is an explicit LLM-based personalization-level classifier (non/partial/personalized) plus fusion weights selected per level on one iKAT collection and transferred to the other. On its face the results are strong: APCIR beats the cited SOTA on both iKAT-23 and iKAT-24, and the aligned comparisons with fixed retriever/reranker are the right way to handle backbone confounds. The ablations, model-variation experiments, and released code are all to their credit. I'd send this to a serious referee.\n\nThe soft spot is the one the stress-test flags. Nothing in the experiments isolates the per-level conditioning. Algorithm 1 optimizes three weight triples, one per level, but Table 2's baselines (Equal, Random, Entropy, DEPS) are not allowed to optimize a single global triple on the same validation data. The reported triples within each dataset are close: on iKAT-24 they are (0.20,0.38,0.42), (0.28,0.36,0.36), (0.23,0.40,0.37). So the gains over Equal Personalization might come almost entirely from tuning weights on labeled data, not from adapting to the level. Given the LLM-human agreement on the non-personalized class is 47.8% on iKAT-23, the level signal is noisy and the three triples could be three noisy estimates of the same optimum. The paper's explanation that the gap is annotation discrepancy in the datasets is plausible but untested.\n\nAlso, M in Eq. (5) is never specified. If the grid search optimizes NDCG@3, the MRR and Recall claims are not the optimized objective. The 'training-free' label is generous: selecting weights on one labeled collection and applying them to another is tuning.\n\nNone of this invalidates the headline empirical result that fused reformulated queries with per-level weights beat the compared methods. But it does mean the 'adaptive' mechanism may not be the cause. The fix is straightforward: add a control that grid-searches one global weight triple on the same validation queries, and report which metric M is being optimized. If a single triple does as well, the paper should be reframed as a tuned-fusion method with level detection as a secondary contribution.\n\nWho is this for? Researchers working on conversational IR and iKAT who want a strong, reproducible pipeline and a fair aligned comparison. It deserves peer review, but I would not accept the mechanism claim as established without the missing ablation.","headline":"A useful adaptive-personalization pipeline with consistent iKAT gains, but the paper never shows the per-level fusion weights beat one globally tuned triple, so the core mechanism is under-supported.","tokens_in":21251,"tokens_out":2222,"would_cite":true,"duration_ms":22281,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conversational search improves when an LLM first labels each query turn as non-personalized, partially personalized, or personalized, and the system fuses personalized and neutral retrieval lists with weights tuned per level.","keywords":["conversational information retrieval","adaptive personalization","personalization level identification","query reformulation","ranking fusion","TREC iKAT","large language models","pseudo-response expansion"],"falsifier":"Use the human personalization annotations available in the TREC iKAT data as the level signal in place of the LLM's labels and compare fused results; if human-labeled levels substantially outperform LLM-labeled levels, the level classifier itself is a bottleneck, and if they do not, the paper's claim that the gap reflects annotation noise is supported.","tokens_in":20124,"feed_emoji":"🔎","tokens_out":8426,"duration_ms":76759,"temperature":0.7,"pith_summary":"The paper tries to establish that conversational information retrieval should not personalize every query turn the same way. Its APCIR system first asks an LLM to label each turn as non-personalized, partially personalized, or fully personalized, using in-context examples and step-by-step reasoning; it then rewrites the turn into several query variants, including a personalized query with a pseudo answer and a neutral query with a pseudo answer. A personalization-aware ranking fusion step combines the retrieval lists from these variants with weights that are optimized separately for each personalization level on validation turns and then applied to test turns. On the TREC iKAT 2023 and 2024 test collections, the paper reports consistent gains over prior systems, including cases where the method beats human-written query rewrites. The practical claim is that explicit, per-turn control of how much personalization to apply is better than relying on an LLM to implicitly decide inside a single prompt.","feed_headline":"Adaptive per-query personalization lifts search MRR by up to 39 points","feed_subtitle":"System judges each turn's personalization need, then blends personalized and neutral results with per-level weights.","key_machinery":"The load-bearing mechanism is a personalization-aware ranking fusion controlled by an explicit personalization-level signal. An LLM labels each turn as one of three levels (non-personalized, partially personalized, or fully personalized) using chain-of-thought prompting and in-context examples; the same LLM then generates a personalized reformulation plus pseudo response and a non-personalized reformulation plus pseudo response. After min-max normalization of each retrieval list, the fusion weights for each level are chosen by grid search as the triple that maximizes an evaluation metric over validation turns of that level, subject to the weights summing to one, and the chosen triple is applied to test turns of the same predicted level (Eq. 5, Algorithm 1). The per-level weights are the mechanism that lets the system down-weight personalized results when a turn is predicted not to need them, which is what avoids over-personalization.","core_discovery":"APCIR's central discovery is that personalized retrieval improves when the amount of personalization is decided per query turn and the decision is used to control fusion, not just prompt content. The paper shows in a preliminary experiment that a fully personalized LLM rewrite alone does not beat a non-personalized rewrite, but fusing the two lists does, and that even ground-truth binary selection of profile sentences does not beat fusion. The method therefore separates three levels of personalization, generates both personalized and non-personalized query variants with pseudo responses, and determines, for each level, the weight triple that maximizes a retrieval metric on a validation set (Eq. 5). Applying those level-specific weights at test time yields the reported gains: MRR 57.3 vs 45.7 on iKAT-23 and 82.7 vs 43.2 on iKAT-24 in the open comparison, with the framework also outperforming the compared baselines when retriever and re-ranker are held fixed across methods.","pith_inferences":["If per-level weights generalize across collections, the same three-level taxonomy could be applied to other conversational tasks that mix personal and factual turns; one test is tuning weights on one corpus and transferring to a new domain.","Because the level signal is noisy (77.8%/47.8% and 79.6%/63.0% overlap with human judgments), a calibrated confidence score from the LLM could allow soft fusion weights rather than a hard per-level triple, which the paper leaves implicit.","Equation (5) optimizes weights against a validation metric, which is close to hyperparameter tuning; the sensitivity of the reported gains to the size and composition of the validation query set is not tested in the paper and would be a direct extension.","The finding that annotated personal information does not beat fusion suggests that retrieval-time blending may be more robust than query-rewrite-time filtering; a testable extension would be applying the same fusion idea to other rewrite-based retrieval pipelines."],"forward_implications":["Systems can avoid over-personalization by assigning a low fusion weight to personalized rewrites when the predicted level is non-personalized; the optimized weights differ across levels.","The adaptive mechanism transfers across backbone LLMs and retrievers: gains hold with GPT-4o, ChatGPT-3.5, LLaMA-3.1-8B and Mistral-2-7B, and with BM25, ANCE and SPLADE.","First-stage fusion can matter more than re-ranking: on iKAT-23, applying re-rankers to SPLADE runs degraded several systems, including APCIR, suggesting the fused list is already strong.","APCIR can exceed human-written query rewrites on iKAT-24, indicating that explicit level detection plus fusion can substitute for costly manual reformulation in personalized CIR."],"supporting_citations":[{"why":"Supplies the TREC iKAT 2024 test collection with its user profiles, used for evaluation and weight-transfer experiments.","marker":"[2]"},{"why":"Supplies the TREC iKAT 2023 dataset, its user profiles, and the human personalization annotations used in the overlap analysis.","marker":"[4]"},{"why":"Provides the PCIR baseline that performs personalized query reformulation with in-context learning, which APCIR is positioned against and extends.","marker":"[51]"},{"why":"Provides the MQ4CS baseline that generates multiple query reformulations and aggregates ranking lists, the closest multi-query competitor.","marker":"[29]"},{"why":"Provides the generate-then-retrieve baseline that uses LLM-generated answers to ground personalized query reformulation.","marker":"[1]"},{"why":"Supplies the RAR prompting setup the paper adapts for query reformulation and pseudo-response generation.","marker":"[32]"},{"why":"Motivates the query-expansion idea of appending LLM-generated pseudo responses to reformulated queries.","marker":"[68]"},{"why":"Provides the SPLADE-v3 retriever used in the main APCIR configuration for sparse-dense retrieval.","marker":"[25]"},{"why":"Provides the monoT5 re-ranker used in the aligned comparison and re-ranking experiments.","marker":"[52]"},{"why":"Supplies evidence that excessive personalization hurts search performance, which motivates the adaptive level design.","marker":"[13]"}],"fun_headline_variants":["Adaptive personalization beats one-size-fits-all retrieval","Per-turn personalization boosts conversational search","Fusion of personalized and neutral queries wins retrieval","On-demand personalization lifts conversational IR","Query-level personalization levels improve retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that queries sharing an LLM-predicted personalization level are similar enough across datasets that one weight triple per level, tuned on a validation collection, transfers to the test collection.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive personalization beats one-size-fits-all retrieval","Per-turn personalization boosts conversational search","Fusion of personalized and neutral queries wins retrieval","On-demand personalization lifts conversational IR","Query-level personalization levels improve retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1213,"prompt_tokens":928,"completion_tokens":285,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":544,"tokens_out":285,"duration_ms":3650,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:33:59.263240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the human personalization annotations available in the TREC iKAT data as the level signal in place of the LLM's labels and compare fused results; if human-labeled levels substantially outperform LLM-labeled levels, the level classifier itself is a bottleneck, and if they do not, the paper's claim that the gap reflects annotation noise is supported.","supporting_citations":[{"cited_title":"History-Aware Conversational Dense Retrieval","cited_arxiv_id":"2401.16659","evidence_quote":"Provides the PCIR baseline that performs personalized query reformulation with in-context learning, which APCIR is positioned against and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RAR prompting setup the paper adapts for query reformulation and pseudo-response generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the query-expansion idea of appending LLM-generated pseudo responses to reformulated queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies evidence that excessive personalization hurts search performance, which motivates the adaptive level design."}],"review_version":2}