REVIEW 3 major objections 4 minor
Can Transformers Break Encryption Schemes via In-Context Learning?
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Transformers can infer hidden substitution rules from a few plaintext–ciphertext pairs and decode new ciphertext words, extending in-context learning to discrete cryptographic mappings.
desk verdict A clear ICL benchmark proposal for classic ciphers, but the abstract alone cannot support the title's claim about breaking encryption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The hidden bijective mapping between ciphertext and plaintext characters is the central object. In the monoalphabetic case it is a single permutation of the alphabet; in the Vigenère case it is a position-dependent set of shifts that can also be seen as a permutation over (position, character) pairs. The in-context demonstration pairs are the mechanism by which the model must constrain this mapping, and the new ciphertext word is the probe that tests whether the inferred rule generalizes.
What would settle it
If a transformer is shown a randomly chosen substitution cipher with a few matching pairs and a new ciphertext word that shares no character n-grams with the demonstrations, and it fails to decode it correctly, then the claim that it infers the general mapping is not supported.
Extended reading notes
Core claim
The central claim is that transformers can, in a single forward pass, learn a hidden substitution cipher from a few in-context examples and generalize to decode unseen words. For monoalphabetic substitution, the hidden rule is a permutation of the alphabet; for Vigenère, it is a set of Caesar shifts controlled by a repeating key. The paper treats success at this task as evidence that in-context learning operates over discrete bijective functions, not just continuous or symbolic well-structured functions studied previously. The demonstration pairs are the only supervision signal, and the evaluation is whether the model can apply the inferred rule to a novel ciphertext word.
Load-bearing premise
The assumption that a small number of demonstration pairs sufficiently constrains the hidden substitution so that decoding a new word actually tests rule inference rather than pattern matching or leakage.
Editorial extensions
If this is right
- Demonstrates in-context learning over discrete bijective mappings, broadening the ICL literature beyond continuous function classes.
- Suggests that transformers can perform rudimentary cryptanalysis on classical ciphers directly from context, without any fine-tuning.
- Provides a benchmark task that isolates rule inference from pattern matching, useful for studying inductive biases in sequence models.
- Opens the possibility of probing whether models can infer other structured discrete rules, such as transpositions or keyed ciphers, with only a few examples.
Reading between the lines
- The paper's setup could be extended to test whether success depends on the language prior of the plaintext; decoding random-looking plaintext words would help separate rule inference from frequency-based guesswork.
- A natural next step is to vary the number of demonstration pairs and measure how many are needed to uniquely determine the substitution, giving a measure of sample efficiency for rule learning.
- The same framework could be applied to affine ciphers or generalized substitution systems, where the hidden mapping has a parameterized structure, to see whether transformers discover the parametric form.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes applying in-context learning (ICL) to classical ciphers: monoalphabetic substitution and Vigenère. The abstract describes a task in which a transformer receives a small set of (ciphertext, plaintext) demonstration pairs and must infer the underlying substitution to decode a new ciphertext word. The submission frames this as a novel structured inference challenge for evaluating transformers' inductive biases. However, the manuscript contains only an abstract and no full text, experimental results, baselines, or control analyses.
Significance. If properly demonstrated, the proposed task would extend ICL research from continuous function classes to discrete bijective mappings, providing a useful stress test for shortcut learning and for understanding the limits of transformer generalization. The authors correctly identify an interesting problem and provide a code link, which is a useful starting point. However, the central empirical claim—that transformers can actually perform this task—is entirely unsupported in the submitted manuscript. The contribution currently stands as a proposal, not a validated result.
major comments (3)
- [Abstract, paragraph 2] The manuscript reports no experimental results, baselines, error bars, or control experiments. The central claim that transformers can infer the substitution and decode a new word is an assertion, not a demonstrated finding. Without any evaluation, the paper cannot support its title's question. This is load-bearing because the entire contribution is empirical.
- [Abstract, paragraph 2] The task specification is underdetermined, making the ICL claim unauditable. To attribute decoding success to genuine inference of the hidden bijection, the demonstration pairs must be the only channel of information about the specific permutation, and test words must not be decodable by shortcuts such as substring matching with demonstrations, exploiting language-model priors (especially for Vigenère, which preserves letter statistics), or retrieval if test words appear in context. The abstract does not specify how demonstrations are sampled, how test words are selected, or whether they are disjoint from the demonstration set. If test words share characters with demonstrations, success may reflect partial mapping plus priors; if not, the mapping may be underdetermined. These controls are essential to validate the headline capability.
- [Abstract, paragraph 1] Calling monoalphabetic substitution and Vigenère 'private-key encryption schemes' overstates their status; they are classical ciphers and not modern private-key encryption. The phrase 'break encryption schemes' in the title is a far stronger claim than the proposed setting, which only tests decoding of a single word from known demonstration pairs. The relationship between this task and actual encryption/security is not justified.
minor comments (4)
- [Title and Abstract] The title is a yes/no question, but the abstract does not provide an answer. Rephrase to reflect the proposal or include a preliminary finding.
- [Abstract, paragraph 2] The phrase 'small set' is vague. Specify the number of demonstration pairs and the alphabet size; this is a key parameter for underdetermination.
- [Code availability] The code link is mentioned, but the manuscript gives no details on model architecture, training data, or evaluation protocol. For reproducibility, these should be included.
- [Related work] The abstract mentions prior ICL work on linear functions, decision trees, and neural networks but does not cite or discuss work on ICL for discrete symbolic mappings, algorithmic reasoning, or cryptographic learning. A proper related-work section is needed.
Circularity Check
No circular derivation visible in the abstract; under-specification is a validity risk, not circularity.
full rationale
The available text is only the abstract. There are no equations, no fitted parameters, no self-citations, and no uniqueness theorems invoked. The central claim—that a transformer can infer a hidden substitution from demonstration pairs and decode a new ciphertext word—is an empirical capability claim, not a derivation that reduces to its own inputs. The abstract does not specify how demonstration pairs are sampled or how test words are chosen, so the risk that the task collapses to retrieval or language-prior leakage is a correctness/experimental-design concern, not a circularity concern. Per the hard rules, under-specification and non-standard evaluation are not circularity arguments. Therefore the appropriate score is 0.
Assumptions & free parameters
free parameters (1)
- in-context demonstration set size =
not specified (described as 'a small set')
assumptions (4)
- domain assumption Monoalphabetic substitution and Vigenère ciphers are deterministic bijective functions from plaintext to ciphertext over a fixed alphabet.
- domain assumption A small set of labeled pairs suffices in principle to constrain the underlying substitution enough to decode new words.
- domain assumption Transformers' ICL, previously demonstrated for linear functions, decision trees and neural networks, transfers to discrete, symbolic bijections.
- domain assumption The evaluation prevents leakage and memorization between demonstration pairs and test words.
Cite this review
Pith. "Pith review of Can Transformers Break Encryption Schemes via In-Context Learning?." pith.science (2026). https://pith.science/paper/7M2IGUNQ
@misc{pith2026250810235,
author = {Pith},
title = {Pith review of: Can Transformers Break Encryption Schemes via In-Context Learning?},
year = {2026},
howpublished = {\url{https://pith.science/paper/7M2IGUNQ}},
note = {Machine review of arXiv:2508.10235}
}
read the original abstract
In-context learning (ICL) has emerged as a powerful capability of transformer-based language models, enabling them to perform tasks by conditioning on a small number of examples presented at inference time, without any parameter updates. Prior work has shown that transformers can generalize over simple function classes like linear functions, decision trees, even neural networks, purely from context, focusing on numerical or symbolic reasoning over underlying well-structured functions. Instead, we propose a novel application of ICL into the domain of cryptographic function learning, specifically focusing on ciphers such as mono-alphabetic substitution and Vigen\`ere ciphers, two classes of private-key encryption schemes. These ciphers involve a fixed but hidden bijective mapping between plain text and cipher text characters. Given a small set of (cipher text, plain text) pairs, the goal is for the model to infer the underlying substitution and decode a new cipher text word. This setting poses a structured inference challenge, which is well-suited for evaluating the inductive biases and generalization capabilities of transformers under the ICL paradigm. Code is available at https://github.com/adistomar/CS182-project.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.