{"id":"ea688687-ffa0-4d52-9085-7d6e61575302","arxiv_id":"2508.10235","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper proposes using in-context learning to infer monoalphabetic substitution and Vigenère cipher mappings from a few plaintext-ciphertext pairs and decode new encoded words.","lead":"This paper tests whether transformers can learn classical ciphers from a few matching examples and decode new encoded words. The result would clarify how in-context learning generalizes to discrete, structured rules, though not to modern encryption.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract under-specifies how the task prevents leakage and tests genuine rule inference, so the central ICL claim is not yet auditable.","rationale":"The reader's verdict of UNVERDICTED is well-justified by the absence of the full text. Our stress test identifies a specific, load-bearing condition that the abstract leaves unspecified: the task design must rule out shortcuts so that success genuinely reflects in-context inference of a hidden bijection. This is the same concern the reader phrased as underdetermination and leakage, so there is no disagreement. Since the full text is unavailable, the verdict stays UNVERDICTED; the concrete test would, if the paper were available, settle whether the concern actually lands.","tokens_in":695,"tokens_out":1452,"duration_ms":16338,"concrete_test":"Inspect the data-generation code in the linked repo (adistomar/CS182-project) and run an ablation where test words are restricted to characters that never appear in the demonstration pairs. If accuracy falls to chance, success on the original task is due to per-character lookup or language priors, not ICL of the bijection. Also verify that the test ciphertext words do not appear anywhere in the context sequences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that transformers can infer a hidden substitution from a small set of (cipher, plain) pairs and decode new words. For this to test ICL of the bijection, two conditions must hold: (i) the demonstration pairs must be the only channel carrying information about the specific cipher permutation; (ii) the new ciphertext word must not be decodable by an alternative shortcut—e.g., matching substrings from the demonstrations, exploiting language-model priors over plaintext (since both ciphers preserve some letter statistics, Vigenère especially), or copying the answer if the test word appears in the context. The abstract states neither how demonstration pairs are sampled nor how test words are selected. If test words are not disjoint from the demonstration set, or if characters in the test word never appear in demonstrations, the task either collapses to retrieval or demands a language prior rather than a learned bijection. Without this control, the headline capability claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes applying in-context learning (ICL) to classical ciphers: monoalphabetic substitution and Vigenère. The abstract describes a task in which a transformer receives a small set of (ciphertext, plaintext) demonstration pairs and must infer the underlying substitution to decode a new ciphertext word. The submission frames this as a novel structured inference challenge for evaluating transformers' inductive biases. However, the manuscript contains only an abstract and no full text, experimental results, baselines, or control analyses.","tokens_in":924,"tokens_out":3536,"duration_ms":36574,"significance":"If properly demonstrated, the proposed task would extend ICL research from continuous function classes to discrete bijective mappings, providing a useful stress test for shortcut learning and for understanding the limits of transformer generalization. The authors correctly identify an interesting problem and provide a code link, which is a useful starting point. However, the central empirical claim—that transformers can actually perform this task—is entirely unsupported in the submitted manuscript. The contribution currently stands as a proposal, not a validated result.","major_comments":[{"comment":"The manuscript reports no experimental results, baselines, error bars, or control experiments. The central claim that transformers can infer the substitution and decode a new word is an assertion, not a demonstrated finding. Without any evaluation, the paper cannot support its title's question. This is load-bearing because the entire contribution is empirical.","section":"Abstract, paragraph 2"},{"comment":"The task specification is underdetermined, making the ICL claim unauditable. To attribute decoding success to genuine inference of the hidden bijection, the demonstration pairs must be the only channel of information about the specific permutation, and test words must not be decodable by shortcuts such as substring matching with demonstrations, exploiting language-model priors (especially for Vigenère, which preserves letter statistics), or retrieval if test words appear in context. The abstract does not specify how demonstrations are sampled, how test words are selected, or whether they are disjoint from the demonstration set. If test words share characters with demonstrations, success may reflect partial mapping plus priors; if not, the mapping may be underdetermined. These controls are essential to validate the headline capability.","section":"Abstract, paragraph 2"},{"comment":"Calling monoalphabetic substitution and Vigenère 'private-key encryption schemes' overstates their status; they are classical ciphers and not modern private-key encryption. The phrase 'break encryption schemes' in the title is a far stronger claim than the proposed setting, which only tests decoding of a single word from known demonstration pairs. The relationship between this task and actual encryption/security is not justified.","section":"Abstract, paragraph 1"}],"minor_comments":[{"comment":"The title is a yes/no question, but the abstract does not provide an answer. Rephrase to reflect the proposal or include a preliminary finding.","section":"Title and Abstract"},{"comment":"The phrase 'small set' is vague. Specify the number of demonstration pairs and the alphabet size; this is a key parameter for underdetermination.","section":"Abstract, paragraph 2"},{"comment":"The code link is mentioned, but the manuscript gives no details on model architecture, training data, or evaluation protocol. For reproducibility, these should be included.","section":"Code availability"},{"comment":"The abstract mentions prior ICL work on linear functions, decision trees, and neural networks but does not cite or discuss work on ICL for discrete symbolic mappings, algorithmic reasoning, or cryptographic learning. A proper related-work section is needed.","section":"Related work"}],"recommendation":"major_revision","confidential_remarks":"The submission as provided appears to be an abstract only, with no full text. This may warrant a desk rejection at the editor's discretion. I recommend major revision rather than outright rejection because the proposed task is interesting and the deficiencies, while substantial, are addressable through a complete experimental study with rigorous controls. The leakage/underdetermination concern is especially critical: without explicit controls on demonstration and test word selection, any positive result would be ambiguous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this one from the abstract alone: the paper proposes using in-context learning to infer monoalphabetic substitution and Vigenère mappings from a few (cipher, plain) pairs, then decode new words. That task is new for ICL, and it is a reasonable way to probe whether transformers can learn discrete bijections in context. The authors cite the standard ICL results (linear functions, decision trees) and correctly frame this as an extension rather than a new mechanism. Credit where due: the problem is clearly stated, the ciphers are precisely defined, and they promise code. As a benchmark idea, it could be useful for studying rule inference versus pattern matching in transformers.\n\nThe soft spots are mostly what is missing. The abstract reports no results, no baselines, no error bars, no controls. The title says \"break encryption schemes,\" which is overstated for ciphers that are already trivially broken by frequency analysis; the real value is as an ICL capability probe, not a security threat. The stress-test concern about leakage is legitimate. For the task to test genuine rule inference, the demonstration pairs must be the only channel carrying the specific permutation, and the test word must not be retrievable by copying, substring matching, or language-model priors over plaintext. The abstract says nothing about how demonstrations are sampled or how test words are chosen, so the central claim is not yet auditable. That said, the absence of these details in an abstract is not fatal if the full text addresses them. I could not see the full text, so I cannot judge the actual experiments.\n\nMy take: this is a legitimate proposal that deserves a proper referee if the full paper exists and contains controlled experiments. The authors seem to know what they are doing; the abstract is coherent and the task is well-motivated. It is not a breakthrough, and the title should be toned down, but it is a fair candidate for a workshop-style paper or a short conference submission. I would not cite it without seeing results, and I would not bring it to a reading group until the full text is available. But a serious editor should send it to peer review rather than desk-reject, because the benchmark idea could be valuable to the ICL community and the paper appears to be in good faith.","headline":"A clear ICL benchmark proposal for classic ciphers, but the abstract alone cannot support the title's claim about breaking encryption.","tokens_in":1348,"tokens_out":1055,"would_cite":false,"duration_ms":11969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformers can infer hidden substitution rules from a few plaintext–ciphertext pairs and decode new ciphertext words, extending in-context learning to discrete cryptographic mappings.","keywords":["in-context learning","transformers","cryptography","substitution ciphers","Vigenère cipher","discrete bijective mappings","inductive bias","generalization"],"falsifier":"If a transformer is shown a randomly chosen substitution cipher with a few matching pairs and a new ciphertext word that shares no character n-grams with the demonstrations, and it fails to decode it correctly, then the claim that it infers the general mapping is not supported.","tokens_in":631,"feed_emoji":"🔐","tokens_out":8120,"duration_ms":62299,"temperature":0.7,"pith_summary":"The paper proposes applying in-context learning to classical ciphers, specifically monoalphabetic substitution and Vigenère ciphers, where a fixed but hidden bijective mapping connects plaintext and ciphertext characters. The goal is for a transformer to infer this mapping from a small set of (ciphertext, plaintext) demonstration pairs and then decode a new ciphertext word. This extends prior in-context learning results beyond continuous function classes to discrete, structured, secret mappings, and tests whether the model performs rule inference rather than pattern matching. The authors argue this is a well-suited structured inference challenge for probing the inductive biases and generalization capabilities of transformers.","feed_headline":"Decode unseen words: transformers learn ciphers from a few pairs","feed_subtitle":"With only a few (cipher, plain) pairs, transformers infer the hidden mapping and decode new words.","key_machinery":"The hidden bijective mapping between ciphertext and plaintext characters is the central object. In the monoalphabetic case it is a single permutation of the alphabet; in the Vigenère case it is a position-dependent set of shifts that can also be seen as a permutation over (position, character) pairs. The in-context demonstration pairs are the mechanism by which the model must constrain this mapping, and the new ciphertext word is the probe that tests whether the inferred rule generalizes.","core_discovery":"The central claim is that transformers can, in a single forward pass, learn a hidden substitution cipher from a few in-context examples and generalize to decode unseen words. For monoalphabetic substitution, the hidden rule is a permutation of the alphabet; for Vigenère, it is a set of Caesar shifts controlled by a repeating key. The paper treats success at this task as evidence that in-context learning operates over discrete bijective functions, not just continuous or symbolic well-structured functions studied previously. The demonstration pairs are the only supervision signal, and the evaluation is whether the model can apply the inferred rule to a novel ciphertext word.","pith_inferences":["The paper's setup could be extended to test whether success depends on the language prior of the plaintext; decoding random-looking plaintext words would help separate rule inference from frequency-based guesswork.","A natural next step is to vary the number of demonstration pairs and measure how many are needed to uniquely determine the substitution, giving a measure of sample efficiency for rule learning.","The same framework could be applied to affine ciphers or generalized substitution systems, where the hidden mapping has a parameterized structure, to see whether transformers discover the parametric form."],"forward_implications":["Demonstrates in-context learning over discrete bijective mappings, broadening the ICL literature beyond continuous function classes.","Suggests that transformers can perform rudimentary cryptanalysis on classical ciphers directly from context, without any fine-tuning.","Provides a benchmark task that isolates rule inference from pattern matching, useful for studying inductive biases in sequence models.","Opens the possibility of probing whether models can infer other structured discrete rules, such as transpositions or keyed ciphers, with only a few examples."],"supporting_citations":[],"fun_headline_variants":["Transformers learn substitution ciphers from a few examples","In-context learning lets transformers decode unseen ciphers","Transformers infer hidden cipher mappings from few pairs","Transformers crack classical ciphers from few pairs"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The assumption that a small number of demonstration pairs sufficiently constrains the hidden substitution so that decoding a new word actually tests rule inference rather than pattern matching or leakage.","fun_headline_variants_meta":{"raw":{"variants":["Transformers learn substitution ciphers from a few examples","In-context learning lets transformers decode unseen ciphers","Transformers infer hidden cipher mappings from few pairs","Transformers crack classical ciphers from few pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000871,"raw_usage":{"total_tokens":3593,"prompt_tokens":713,"completion_tokens":2880,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2825}},"tokens_in":457,"tokens_out":2880,"duration_ms":20804,"temperature":1.0,"reasoning_tokens":2825,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:33:59.411929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a transformer is shown a randomly chosen substitution cipher with a few matching pairs and a new ciphertext word that shares no character n-grams with the demonstrations, and it fails to decode it correctly, then the claim that it infers the general mapping is not supported.","supporting_citations":[],"review_version":1}