{"id":"0d7a0a77-2758-4bee-8e9f-5d0f0d24888c","arxiv_id":"2504.19395","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs perform consistently better on tasks where input words are replaced with a consistent, reversible substitution cipher than when replacements are random, and the authors propose this gap as a measure of task learning in in-context learning.","lead":"This paper measures whether large language models can learn from examples by replacing some words with a secret code and checking if they still solve the task. The authors find models do slightly better when the code is consistent and reversible, and they use that difference as a proxy for true in-context learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NON-BIJECTIVE baseline confounds irreversibility with per-occurrence randomness, so the claimed 'only difference' in Section 2.5 is false; the reported gap may measure text chaos rather than task learning. A fixed many-to-one control condition would settle this.","rationale":"The reader's weakest assumption identifies essentially the same load-bearing concern: the NON-BIJECTIVE baseline is more chaotic and has less repetition, so the gap may reflect processing difficulty rather than task learning. My stress-test sharpens this by pointing to an internal inconsistency in the paper's own framing: Section 2.5 claims the two conditions differ only in their token mapping functions, but Section 2.4 defines NON-BIJECTIVE as an independent random draw per occurrence, making it a stochastic process rather than a fixed cipher. This is not merely a missing nicety; it directly undermines the central inference that the gap is a measure of task learning. The proposed fixed many-to-one control is the minimal intervention that isolates injectivity/reversibility from consistency and text-level randomness. If the gap persists against this control, the authors' interpretation is substantially vindicated; if it collapses, the central claim must be weakened to say that the gap measures consistent token remapping, which is a different and less specific statement. I therefore do not recommend changing the reader's CONDITIONAL verdict: the paper presents a useful new measurement idea and transparently discloses limitations, but it needs this control before the measure can be adopted as a quantification of task learning. My agreement with the reader is 'agree' because the reader explicitly suggested the fixed many-to-one substitution as the missing control and identified the same underlying confound.","tokens_in":25843,"tokens_out":8184,"duration_ms":89509,"concrete_test":"Recompute the Table 1 protocol (20-shot, same four datasets, Llama-3.1-8B) while adding a FIXED-NON-INJECTIVE condition: choose a deterministic many-to-one mapping h:S->S with |h(S)| approximately |S|/2, apply it consistently to every occurrence, and keep the same shuffle rate, Zipfian frequency grouping, priority sampling, and prompt template as in the BIJECTIVE condition. Compare three-way accuracy: BIJECTIVE, FIXED-NON-INJECTIVE, and the current random NON-BIJECTIVE baseline. If BIJECTIVE is not significantly better than FIXED-NON-INJECTIVE, the reported gap is attributable to consistency/randomness rather than reversibility, and the measure needs reinterpretation. If BIJECTIVE significantly outperforms FIXED-NON-INJECTIVE, the reversibility property is doing real work and the central interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the accuracy gap between BIJECTIVE and NON-BIJECTIVE ciphers quantifies in-context task learning. Section 2.5 states that 'the only difference between the two is their token mapping functions: BIJECTIVE cipher mapping allows a reversible mapping... NON-BIJECTIVE cipher removes the learnable patterns.' But as defined in Section 2.4, the NON-BIJECTIVE condition is not a fixed mapping at all: each occurrence of a ciphered token is independently replaced by a uniformly random token from S. This introduces a second, uncontrolled difference beyond injectivity: the BIJECTIVE condition is deterministic and consistent across occurrences, while the NON-BIJECTIVE condition is stochastic per occurrence. Consequently, NON-BIJECTIVE ciphered text has fewer repeated tokens, a higher type-token ratio, and generally more chaotic n-gram statistics. An LLM may perform worse on such text simply because it is harder to process, not because the task is not learnable. The claimed measurement of 'learning' is therefore not yet isolated: the gap in Table 1 could be driven by language-model fluency, lexical priming from priority sampling (Section 3.1), or the ability to exploit consistent token-label co-occurrence in the demonstrations, rather than by deciphering a reversible cipher. This is a load-bearing correctness risk because the paper's central interpretive claim depends on the two conditions differing only in reversibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ICL CIPHERS, a token-level substitution cipher applied to in-context learning inputs, with two conditions: a BIJECTIVE cipher using a fixed one-to-one token mapping, and a NON-BIJECTIVE baseline in which each occurrence of a ciphered token is replaced by an independently sampled token. The authors report that LLMs achieve higher accuracy on BIJECTIVE-ciphered prompts than on NON-BIJECTIVE-ciphered prompts across six models and four datasets, and they interpret this accuracy gap as a quantitative measure of task learning (TL) rather than task retrieval (TR). They also analyze the effect of shuffle rate, number of demonstrations, alignment, model size, and grammatical roles, and they present a Logit Lens probing analysis suggesting that representations shift toward substituted tokens under the BIJECTIVE condition. The central claim is that the BIJECTIVE-minus-NON-BIJECTIVE gap isolates and quantifies inference-time task learning.","tokens_in":26138,"tokens_out":3978,"duration_ms":40960,"significance":"The paper targets a central open question in ICL research and proposes a novel, general reformulation framework that is easy to apply to arbitrary tasks. The empirical scope is broad (six models, four datasets, multiple shot counts) and the authors release code and report paired significance tests, which is a strength. If the gap truly isolated task learning, the method would be an important tool for probing ICL. However, the core interpretive claim rests on the assumption that the two cipher conditions differ only in reversibility, and that assumption is not satisfied by the current design, so the empirical gap may at least partly reflect low-level text processing difficulty rather than task learning. The probing analysis is suggestive but, as the authors acknowledge, weak. The paper would be substantially stronger with a control condition that keeps per-occurrence consistency while removing injectivity, plus a more careful treatment of null results and rate-selection choices.","major_comments":[{"comment":"The central measurement assumption is confounded. Section 2.4 defines the NON-BIJECTIVE cipher so that each occurrence of a token is replaced by a uniformly random token from S, whereas the BIJECTIVE cipher uses a fixed mapping so that repeated occurrences are replaced consistently. Section 2.5 then claims that \"the only difference between the two is their token mapping functions.\" That claim is false: the two conditions also differ in per-occurrence consistency, repetition statistics, and overall n-gram coherence. The NON-BIJECTIVE ciphered text is therefore more chaotic and harder for an LM to process for reasons unrelated to whether the mapping is learnable. The reported accuracy gap in Tables 1 and 2 could be driven by this processing difficulty rather than by task learning. This is load-bearing for the paper's central claim. I request a control condition that preserves per-token consistency but removes reversibility (e.g., a fixed many-to-one substitution), and ideally an unciphered baseline, to support the interpretation that the gap measures learning.","section":"§2.4 and §2.5"},{"comment":"The claim of a consistent gap across models and datasets is overstated. Of the sixteen comparisons in Table 1, three are not positive improvements: OLMo on WinoGrande (-0.3), Gemma2 on HellaSwag (-0.5), and Gemma2 on WinoGrande (+0.0). In addition, Qwen2.5 on WinoGrande (+1.2) is not statistically significant (p=0.084 in Appendix Table 7). The text acknowledges only Olmo on WinoGrande and Gemma2 on HellaSwag as exceptions, and does not discuss the null WinoGrande results. Furthermore, the shuffle rate is set to a different value for each dataset (SST-2 r=0.5, Amazon r=0.6, HellaSwag r=0.3, WinoGrande r=0.1) without a stated selection rule. This introduces researcher degrees of freedom and makes the cross-dataset consistency claim harder to evaluate. Please report results for a shared set of shuffle rates (or a principled rule) and discuss the negative/null cases explicitly.","section":"Table 1 and §4.1"},{"comment":"The shuffle-rate analysis is reported mainly for Llama-3.1-8B on Amazon (and SST-2 in the appendix), and the choice of the interval where the gap is largest is used to justify the per-dataset rates. Since the central evidence is the gap, the reader needs to know whether the pattern holds across models and datasets at a common set of rates, or whether the chosen rates were selected because they show the effect. Additionally, the priority-sampling comparison in Appendix D covers only two datasets and is summarized qualitatively as \"similar trends but more unstable\"; quantitative results for all datasets would help rule out that the gap is an artifact of the sampling procedure interacting with the NON-BIJECTIVE condition.","section":"§3.1, §4.2, and Appendix D"}],"minor_comments":[{"comment":"Typo: \"disentangling these the two modes\" should read \"disentangling these two modes.\"","section":"Abstract and §1"},{"comment":"Typo: \"multiple occurences of a token are conisistently replaced\" should read \"consistently replaced.\"","section":"§2.4"},{"comment":"Typos: \"Peformance\" appears in the captions of Figures 7 and 8; \"regared\" appears in the captions of Tables 7–10.","section":"Appendix D and G"},{"comment":"The tables would be easier to interpret with confidence intervals or standard deviations for the reported accuracies, since each value is an average over three runs. The McNemar tests are reported without correction for the multiple comparisons across models and datasets.","section":"Tables 1–3"},{"comment":"The probing analysis is described as evidence of deciphering, but the reported rank-difference trends are on a small set of 30 token pairs and the effect sizes are not quantified beyond the figures. A small clarification of how many occurrences and layers are averaged in Figure 5 would improve reproducibility.","section":"§4.7 and Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The core experimental idea is interesting and the paper is well within the scope of the journal, but the central interpretational claim (the gap quantifies task learning) is currently under-supported because the NON-BIJECTIVE baseline is confounded with per-occurrence randomness. I would ask the authors to add a fixed many-to-one control condition and an unciphered baseline; without those, the gap could be explained by low-level processing difficulty. The paper's own Discussion is appropriately cautious, but the abstract and §4.1 make stronger claims that need the additional controls to be defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth a look: using input-side bijective versus non-bijective substitution ciphers to isolate task learning in ICL is a new angle, and the paper's execution is honest. The demo-priority sampling and Zipfian frequency matching show care. But the central contrast is confounded. The NON-BIJECTIVE condition isn't a fixed non-injective mapping; each occurrence of a ciphered token is replaced with an independently sampled token. That changes repetition statistics, type-token ratio, and n-gram fluency, so the gap in Table 1 may partly be a processing-difficulty effect rather than 'learning' in the intended sense. Section 2.5's claim that 'the only difference between the two is their token mapping functions' is therefore inaccurate.\n\nWhat the paper does well: it runs six models over four datasets, reports significance tests, and is transparent about the probing analysis being weak. The finding that BIJECTIVE beats NON-BIJECTIVE in most settings is credible as a phenomenon. The interpretability probe is a nice complement, though the authors themselves downplay it.\n\nWhere it falls short: no unciphered baseline, per-dataset shuffle rates chosen where the effect is visible, and two of the sixteen Table-1 comparisons are null or negative. The gaps are mostly 1 to 10 accuracy points, so practical importance is modest. The bigger issue is the missing control: a fixed many-to-one mapping (e.g., every ciphered token maps to the same token) or a consistent but non-injective mapping would isolate reversibility from per-occurrence randomness. Without that, the 'quantification of TL' claim is premature.\n\nWho this is for: ICL researchers interested in measurement tools. It deserves peer review, but reviewers should ask for the control condition and an unciphered baseline. If the gap survives those, it becomes a useful instrument. As it stands, the paper is a promising proposal rather than a validated measure.","headline":"The bijective-vs-non-bijective cipher gap is a clever but currently confounded measure of ICL task learning; it needs a fixed non-injective control before the 'quantify learning' claim holds.","tokens_in":26701,"tokens_out":2440,"would_cite":false,"duration_ms":24995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs perform better on in-context tasks scrambled with a reversible word cipher than with random word noise, and this accuracy gap is proposed as a practical measure of task learning.","keywords":["in-context learning","task learning","task retrieval","substitution cipher","large language models","few-shot learning","prompt reformulation","logit lens probing"],"falsifier":"Run a third control in which each token is always replaced by the same random token but the mapping is many-to-one (non-injective): if accuracy on this condition matches the bijective cipher, the gap is explained by consistency alone and would not measure learnability; if it matches the non-bijective condition, the paper's interpretation is supported.","tokens_in":25651,"feed_emoji":"🔐","tokens_out":9344,"duration_ms":80739,"temperature":0.7,"pith_summary":"This paper tries to separate two things that are tangled inside in-context learning: the model retrieving a task it recognizes from pretraining, and the model actually learning a new input-label pattern from the demonstrations on the fly. To isolate the learning component, the authors route standard benchmarks through token-level substitution ciphers, swapping a fraction of the words in each prompt for unrelated words according to a fixed one-to-one key. They compare this against a control where each substituted word is replaced by a randomly sampled word, so no consistent pattern exists. The result is a small but consistent accuracy advantage for the reversible cipher across four datasets and six models, which the paper offers as a practical measure of task learning. A probing analysis adds internal evidence that the model increasingly tracks the ciphered tokens as it sees more demonstrations.","feed_headline":"LLMs learn decipherable ciphers better than gibberish","feed_subtitle":"A small, consistent accuracy gap between reversible and random token shuffles offers a new measure of few-shot learning.","key_machinery":"The load-bearing object is a token-level substitution cipher $c: \\mathcal{V} \\to \\mathcal{V}$ applied to the input side of in-context demonstrations, with a shuffle rate $r$ that fixes how many vocabulary tokens are remapped. The argument rests on the contrast between two mappings over the same token set: a bijective cipher, where every occurrence of a token maps consistently to one paired token and the map is in principle invertible, and a non-bijective cipher, where each occurrence is replaced by an independently sampled token so no consistent mapping exists. Two design choices support the comparison: Zipfian shuffling, which only swaps tokens of similar corpus frequency to reduce frequency confounds, and priority demonstration sampling, which ensures the demonstrations expose the model to the specific substitutions that appear in the test input. The accuracy gap between the two conditions is the paper's proposed measure of task learning.","core_discovery":"The central claim is that LLMs solve in-context tasks whose inputs have been scrambled by a bijective substitution cipher better than they solve the same tasks scrambled by an irreversible random substitution, and that this accuracy gap quantifies task learning. In the bijective condition every occurrence of a chosen token is replaced by the same paired token, so in principle the mapping can be deciphered from enough demonstrations; in the non-bijective condition each occurrence is replaced independently, so there is no pattern to learn. Because the two conditions share the same token set, the same prompts, and the same pipeline, the paper attributes the gap to the model's inference-time ability to learn and apply the latent mapping. The gap is small, often a few accuracy points, but it is consistent across SST-2, Amazon, HellaSwag, and WinoGrande with six models, and it grows with the number of demonstrations before plateauing. Logit Lens probing—decoding internal representations into token probabilities—shows that under the bijective cipher the model's representation increasingly favors the substituted token over the original one as occurrences accumulate, while no such preference develops under random substitution.","pith_inferences":["A control condition with a fixed many-to-one substitution (consistent but non-reversible) would test whether the gap is driven by mapping consistency rather than reversibility; if that control matches the bijective condition, the metric would need reinterpretation.","The same cipher-gap protocol could be applied to non-classification tasks such as arithmetic or structured prediction, where the task-learning component is thought to be larger; whether the gap scales with task complexity is a testable extension.","If the gap truly measures task learning, it should predict transfer: a model showing a large gap on one cipher should adapt faster to an independently generated second cipher, a prediction the paper does not test.","Restricting shuffling to tokens of a single part of speech (as the paper does for nouns) could be extended to control word length, concreteness, or co-occurrence statistics, which would clarify which properties the model exploits when deciphering."],"forward_implications":["The bijective-versus-random accuracy gap gives a task-agnostic diagnostic for when an LLM is learning from demonstrations rather than recognizing a memorized task.","The gap's growth with demonstration count provides the expected scaling signature of a learning signal, so the measure can be used to compare models or training stages.","Since the effect persists in an instruction-tuned model and in a 70B model, the paper's evidence suggests task learning is a general property of current LLMs, not a quirk of one checkpoint.","Ciphering inputs rather than labels sidesteps the need to search pretraining corpora for contamination, making it easier to construct 'novel' task variants from existing benchmarks.","The internal probing results imply that successful models need not fully decode the cipher; capturing task-relevant attributes of ciphered tokens can suffice, which bears on how ICL mechanisms should be modeled."],"supporting_citations":[{"why":"Sets up the task-retrieval vs task-learning distinction and the label-substitution baseline that this paper's input-side cipher is designed to improve upon.","marker":"Pan et al. (2023)"},{"why":"Proposes the dual-operating-modes view of ICL that frames the paper's research question.","marker":"Lin and Lee (2024)"},{"why":"Studies task recognition versus task learning across pretraining, motivating the need for input-space interventions.","marker":"Wang et al. (2024)"},{"why":"Demonstrated that ICL can work with corrupted labels, establishing the 'learning versus content' puzzle the paper addresses.","marker":"Min et al. (2022)"},{"why":"Supplies the statistical test the paper uses to establish that the bijective-minus-non-bijective gap is significant.","marker":"McNemar (1947)"},{"why":"The Logit Lens method used to probe internal representations for evidence of cipher decoding.","marker":"nostalgebraist (2020)"},{"why":"Shows token frequency affects LLM performance, motivating the Zipfian shuffling design.","marker":"Razeghi et al. (2022)"},{"why":"Links performance to token frequency in parametric memory, supporting the frequency-matched shuffling strategy.","marker":"Mallen et al. (2023)"}],"fun_headline_variants":["Reversible ciphers reveal a real learning signal in ICL","LLMs solve reversible cipher tasks better than random shuffles","A small gap between decipherable and gibberish inputs quantifies ICL learning","Cracking the code of in-context learning with substitution ciphers","Measuring task learning in LLMs with a bijective cipher trick"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bijective and non-bijective conditions are assumed to differ only in whether the token mapping is consistent and reversible, so the accuracy gap is interpreted as task learning rather than as a difference in how hard the two kinds of scrambled text are for the language model to process.","fun_headline_variants_meta":{"raw":{"variants":["Reversible ciphers reveal a real learning signal in ICL","LLMs solve reversible cipher tasks better than random shuffles","A small gap between decipherable and gibberish inputs quantifies ICL learning","Cracking the code of in-context learning with substitution ciphers","Measuring task learning in LLMs with a bijective cipher trick"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1522,"prompt_tokens":1048,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":664,"tokens_out":474,"duration_ms":4874,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:53:00.085971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a third control in which each token is always replaced by the same random token but the mapping is many-to-one (non-injective): if accuracy on this condition matches the bijective cipher, the gap is explained by consistency alone and would not measure learnability; if it matches the non-bijective condition, the paper's interpretation is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets up the task-retrieval vs task-learning distinction and the label-substitution baseline that this paper's input-side cipher is designed to improve upon."},{"cited_title":"Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning","cited_arxiv_id":"2406.14022","evidence_quote":"Studies task recognition versus task learning across pretraining, motivating the need for input-space interventions."}],"review_version":1}