{"id":"fe0cc63d-788d-4526-8fb7-d538576de57c","arxiv_id":"2501.09274","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A zero-shot Llama-3.1-8B-Instruct, used as a mutation and crossover operator in directed evolution, outperforms a random-mutation evolutionary algorithm on four of five protein fitness benchmarks.","lead":"This paper tests whether a large language model that was never fine-tuned for biology can propose better protein sequences in an iterative search. On four of five fitness landscapes, the LLM-based search found higher-fitness variants than a random-mutation baseline, but it lost on one benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM gains may reflect parent fitness scores in the prompt rather than LLM protein knowledge; the EA baseline lacks fitness-conditioned proposals, so the central claim is not yet isolated.","rationale":"The reader's weakest_assumption identified exactly the same confound: LLM proposals may reflect conditioning on parent fitness scores or memorized training data rather than online optimization. This is the most load-bearing concern because the central claim is about the LLM as an optimizer, not about the combined system of prompt-plus-fitness. A simple ablation would settle it. The GB1 contradiction is secondary but reinforces that the performance claim is not uniform. Since the paper presents a plausible method but does not isolate the mechanism, conditional acceptance is appropriate; my stress-test does not change that verdict. Agreement is full because the reader and I focus on the same experimental gap, and the proposed concrete test directly addresses it.","tokens_in":10206,"tokens_out":2524,"duration_ms":70906,"concrete_test":"Run the same evolutionary loop on Syn-3bfo and AAV with two modifications: (1) replace the parent fitness scores in the prompt with random numbers or omit them entirely; (2) add a baseline that always mutates the higher-fitness parent by 1-4 random substitutions, followed by the same top-k selection. If LLM performance drops to EA levels under (1) or matches the fitness-conditioned random mutator under (2), the reported advantage is due to fitness conditioning, not the LLM. Also report per-seed GB1 trajectories to verify whether EA truly wins only one seed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 1) is that LLMs themselves can optimize protein fitness zero-shot. But the main comparison in Table 2 pits LLM proposals, which receive the fitness scores of both parent sequences in the prompt (Appendix A.3), against an EA baseline whose mutation and crossover operators are blind to fitness. This is an uneven comparison: the LLM can simply mutate the higher-fitness parent or preserve high-fitness residues, a trivial heuristic that random mutation/crossover cannot exploit. The observed gains on Syn-3bfo, AAV, and GFP could therefore stem from fitness-conditioned proposal generation, not from any protein-specific reasoning or memorized knowledge. No ablation removes fitness scores from the prompt or adds a fitness-conditioned random mutator baseline. Additionally, Table 2 shows EA outperforms the LLM on GB1 at population sizes 32 and 48 (Top1), which directly contradicts the 'much more efficient' claim; the paper's explanation in Section 4.2 ('EA only outperforms one of three random seeds') is not supported by the reported means and standard deviations. These issues leave the central attribution unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-guided directed evolution framework in which Llama-3.1-8B-Instruct proposes new protein variants from pairs of parent sequences, after which the population is rank-selected by fitness. The framework is evaluated on five landscapes (GB1, TrpB, a synthetic SLIP landscape from PDB 3bfo, AAV, and GFP) under single-objective, Hamming-distance-constrained, budget-constrained, and multi-objective settings. The central claim is that a general-purpose LLM can act as a zero-shot mutation/crossover operator, outperforming a standard evolutionary algorithm with random mutation and crossover.","tokens_in":10432,"tokens_out":4478,"duration_ms":43280,"significance":"The paper addresses a timely and practically important question: whether a pretrained LLM can generate useful protein variants without fine-tuning. Its strengths include coverage of three oracle types (exact DMS, synthetic Potts-model, and ML-predicted fitness), a detailed prompt appendix (Appendix A.3), an iteration-count ablation (Table 6), and systematic treatment of constrained and multi-objective optimization. If the main claim were cleanly established, the framework would be a useful addition to ML-guided directed evolution. However, the principal comparison conflates two effects: the LLM receives parent fitness scores in its prompt while the EA baseline receives none, and one benchmark (GB1) shows the EA winning several reported columns. The retrieval-risk from public DMS data is also not addressed. These issues are load-bearing for the claim that the LLM is 'secretly' an optimizer, so the current evidence is suggestive but not yet conclusive.","major_comments":[{"comment":"The prompt shown in Appendix A.3 provides each parent sequence with an explicit fitness value (e.g., 'fitness score: 0.0018'), while the EA baseline receives only the sequence. This gives the LLM a trivial fitness-conditioning advantage: it can edit preferentially from the higher-fitness parent or preserve high-fitness residues, whereas random mutation and crossover cannot exploit fitness at proposal time. The central comparison in Table 2 therefore does not isolate LLM protein knowledge as the source of improvement. Please add an ablation that removes fitness scores from the prompt, and add a fitness-conditioned random baseline (e.g., mutation rate or parent-choice probability proportional to fitness) to test whether the observed gap persists without LLM knowledge.","section":"Appendix A.3, Table 2"},{"comment":"On GB1 with population sizes 32 and 48, Table 2 reports EA Top-1 fitness of 5.38 +/- 1.77 and 4.88 +/- 0.33 versus Ours of 4.34 +/- 0.53 and 4.31 +/- 0.82. This contradicts the introduction's claim that the LLM proposals are 'much more efficient' in the single-objective setting. The explanation in Section 4.2 that 'EA only outperforms one of three random seeds' is not supported by the reported means and standard deviations; with three seeds and overlapping intervals, seed-level trajectories and a statistical test are needed before the cross-dataset claim can be accepted.","section":"Table 2, Section 4.2"},{"comment":"The pseudocode's control flow is ambiguous. When the LLM returns None, the code appends C(s0,s1) but then continues to sample r and may append either M(s0) or 'proposed seq' (which is None) without an explicit guard. When the LLM returns a valid sequence, that sequence is only appended in the 'else' branch after a uniform random draw. As written, it is impossible to determine what fraction of offspring in the reported experiments actually come from the LLM rather than from the random crossover/mutation fallback. Please rewrite the pseudocode and report the average fraction of LLM-generated proposals that enter the offspring set per iteration.","section":"Algorithm 1, Section 3"},{"comment":"GB1, TrpB, AAV, and GFP are public deep-mutational-scanning datasets, and the SLIP landscape is constructed from a public PDB MSA. It is plausible that Llama-3.1's pretraining corpus contains these sequences and associated fitness information, so the model may propose high-fitness variants by retrieval rather than by on-the-fly optimization. As a correctness-risk control, please report how often the LLM's proposed sequences coincide with known high-fitness variants in each benchmark, or evaluate on a synthetic landscape that did not exist before the model's training cutoff. This would make the retrieval risk measurable without requiring a full audit of the pretraining corpus.","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"The sentence that 'the maximum Hamming distance of sequences proposed by EA does not exceed 5' is used to explain identical EA rows for H=5 and H=10, but no evidence is provided. Please report the distribution of edit distances observed for EA proposals or otherwise support this claim.","section":"Section 4.2, Table 4"},{"comment":"In the prompt text, 'minimizing humming distance' should be 'Hamming distance'.","section":"Appendix A.3"},{"comment":"The label 'AA V' should be 'AAV', and the meaning of 'N/A' in the space-size and target-range columns would benefit from a footnote.","section":"Table 1"},{"comment":"The statement that 'linear relationships between positions are less likely in more complex landscapes' is informal; consider reporting an epistasis measure or phrasing this as a testable hypothesis rather than a justification.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the MLDE scope of cs.LG, and the authors have provided useful experimental breadth and prompt transparency. The main risk is attribution: without a fitness-conditioned control and contamination analysis, the reported gains could be explained by prompt conditioning or memorization rather than LLM-based optimization. I would ask for code, seed-level data, and the proposed controls in revision. The relationship to Chen et al. (2024) and Tran & Hy (2024) should also be sharpened, since those works already use LLMs as sequence optimizers; the current text cites them but does not clearly delineate the novel contribution beyond zero-shot prompting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The headline claim—that a general-purpose, zero-shot LLM can act as a mutation/crossover operator for protein engineering—is worth testing, and the paper makes a real attempt. But the current evidence doesn't isolate that claim. The main comparison is uneven: the LLM receives the fitness scores of both parent sequences in its prompt (Appendix A.3), while the EA baseline's mutation and crossover operators are blind to fitness. The LLM can trivially favor the higher-fitness parent or preserve beneficial residues; random mutation can't. That alone could explain most of the gains on Syn-3bfo, AAV, and GFP. The GB1 results also sit awkwardly with the 'much more efficient' claim—EA matches or beats the LLM at Top1 for population sizes 32 and 48. The explanation in Section 4.2 about seeds doesn't line up with the reported means and standard deviations.\n\nWhat's genuinely new: prior work used protein-specific language models or fine-tuned LLMs. This paper runs a standard evolutionary loop with an off-the-shelf instruction-tuned model and shows competitive performance across several fitness landscapes. The breadth of settings—single-objective, constrained, budget-constrained, multi-objective—is a plus, and the prompts are in the appendix, which helps reproducibility.\n\nThe soft spots are the baseline confound above plus an unexamined contamination risk: GB1, AAV, GFP, and TrpB are public DMS datasets that Llama-3.1 may well have seen during pre-training. If the model is retrieving known high-fitness variants, the paper is demonstrating retrieval, not optimization. A simple control—drop fitness scores from the prompt, or add a fitness-conditioned random mutator—would sharpen the attribution. A contamination check with held-out sequences would also help. No code or full prompts for all datasets are released, which makes independent replication harder.\n\nMy take: the paper deserves a serious referee. The core idea is plausible, the experiments are broad, and the issues are fixable. I'd send it to review with a request for the baseline controls and a contamination analysis. Not something I'd cite yet in my own work; worth a reading-group discussion.","headline":"Promising zero-shot LLM evolution, but the baseline compares fitness-conditioned prompts with blind random mutation, so the central claim isn't yet isolated.","tokens_in":10948,"tokens_out":2619,"would_cite":false,"duration_ms":24212,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A general-purpose large language model, used as the mutation and crossover operator in a directed-evolution loop, can propose higher-fitness protein variants on the fly without any fine-tuning.","keywords":["protein sequence optimization","directed evolution","large language models","zero-shot mutation operator","fitness landscape","Pareto optimization","deep mutational scanning","evolutionary algorithm"],"falsifier":"Run the same LLM-guided loop on a protein family that was not present in the LLM's pretraining corpus (for example, a newly designed enzyme with no close public homologs) and compare against random-mutation evolution; if the advantage disappears, the effect is memorization rather than optimization. Alternatively, remove or shuffle the fitness scores in the prompt and check whether proposal quality degrades.","tokens_in":10026,"feed_emoji":"🧬","tokens_out":8873,"duration_ms":73773,"temperature":0.7,"pith_summary":"The paper claims that a general-purpose large language model (Llama-3.1-8B-Instruct), used with no fine-tuning, can act as the mutation and crossover operator inside a directed-evolution loop for protein sequence optimization. The method repeatedly asks the LLM to propose new variants from pairs of parent sequences, scores each proposal with a fitness oracle, and keeps the top-ranked candidates for the next round. Across five fitness landscapes—two exact experimental deep-mutational-scanning datasets (GB1, TrpB), one synthetic Potts-model landscape (Syn-3bfo), and two machine-learning oracles (AAV, GFP)—the LLM-guided search finds higher-fitness candidates than a standard evolutionary algorithm with random mutation and crossover, with the largest advantages on the more nonlinear landscapes. The paper also shows the same loop can obey Hamming-distance budgets, experimental iteration budgets, and multi-objective Pareto targets. If the claim holds, a capable text model can serve as a zero-shot sequence optimizer, potentially accelerating wet-lab directed evolution without training a domain-specific model.","feed_headline":"LLM proposes high-fitness protein variants without fine-tuning","feed_subtitle":"Used as the sequence-proposal step in directed evolution, a generic LLM beats random mutagenesis on five landscapes.","key_machinery":"The central object is the LLM-as-proposer evolutionary loop. The LLM is treated as a black-box stochastic mutation/crossover operator: given a pair of parent sequences—with their fitness scores included in the prompt—it outputs a proposed variant, and if the model returns nothing valid, the algorithm falls back to random crossover or mutation. Selection is elitist top-k on the merged population, and constraints are enforced by rejection sampling plus natural-language instructions in the prompt. What carries the argument is the bias of the LLM's proposals: unlike uniform random edits, they cluster around plausible high-fitness substitutions, and this bias is what makes the search efficient on rugged fitness landscapes.","core_discovery":"On the paper's own terms, the central discovery is that protein fitness optimization does not require a model explicitly trained on protein sequences or fitness labels: an instruction-tuned LLM prompted with the wild-type sequence, two parent sequences, and their fitness scores will propose variants whose fitness distribution is substantially better than random mutation or random recombination. The authors embed this LLM into an evolutionary algorithm—initialize a pool, sample parent pairs, ask the LLM for a crossover or mutation proposal, merge with the old pool, and keep the top-k by oracle fitness—and compare it against a classical evolutionary algorithm under identical population sizes, iterations, initial pools, and selection rules. On the three complex landscapes the LLM version consistently wins; on the near-linear GB1 and TrpB landscapes it is competitive and occasionally loses a seed. The paper further claims the same mechanism works when per-iteration Hamming distance is capped, when the total experimental budget is fixed, and when fitness and Hamming distance are optimized jointly as a two-objective Pareto problem.","pith_inferences":["The paper does not ablate whether the LLM's success comes from reasoning about sequence-function relationships or from retrieving memorized high-fitness variants of these public benchmark proteins; a test on a protein family absent from pretraining data, or with fitness scores stripped from the prompt, would separate the two.","If the mechanism is largely memorization, the method's utility would generalize only to proteins well represented in the LLM's training corpus; if it is genuine in-context reasoning, the same loop could be applied to newly designed or proprietary proteins.","The prompt includes parent fitness values, so a plausible alternative explanation is that the LLM simply interpolates among high-scoring parents; this could be probed by giving the same parents without scores or with shuffled scores.","The framework could be combined with active learning or surrogate-model guidance, where the LLM proposes candidates that the surrogate scores and the experimenter validates, potentially reducing wet-lab rounds further."],"forward_implications":["An off-the-shelf LLM can be dropped into existing directed-evolution pipelines as a proposal generator, with no training or fine-tuning step.","The method transfers across fitness oracles of very different types—exact wet-lab measurements, statistical-energy models, and learned surrogates—so it is not tied to one scoring function.","Experimental budgets can be encoded directly as natural-language constraints (Hamming distance to wild type or to previously tested sequences), and the LLM respects them, reducing wasted experiments.","Fitness and mutational distance can be optimized jointly, with the LLM discovering Pareto-frontier sequences comparable to or better than random evolution.","On rugged, high-dimensional landscapes the advantage over random mutagenesis grows, suggesting LLM proposals are most useful precisely where classical directed evolution stalls."],"supporting_citations":[{"why":"Supplies the GB1 experimental deep-mutational-scanning fitness landscape used as an exact oracle benchmark.","marker":"Wu et al. (2016)"},{"why":"Supplies the TrpB combinatorially complete active-site fitness landscape used as an exact oracle benchmark.","marker":"Johnston et al. (2024)"},{"why":"Supplies the GFP fluorescence deep-mutational-scanning dataset that defines the ML-oracle task.","marker":"Sarkisyan et al. (2016)"},{"why":"Supplies the AAV capsid deep-mutational-scanning dataset used as the second ML-oracle task.","marker":"Bryant et al. (2021)"},{"why":"Defines the SLIP tuned fitness landscape used to construct the synthetic Syn-3bfo oracle.","marker":"Thomas et al. (2022)"},{"why":"Specifies the ML oracle training procedure and the 'medium difficulty' initial-pool selection criteria for AAV and GFP.","marker":"Kirjner et al. (2023)"},{"why":"Represents the fine-tuning-based LLM protein optimization approach that the paper contrasts with zero-shot prompting.","marker":"Chen et al. (2024)"},{"why":"Establishes the ML-guided directed evolution paradigm and the random-mutation baseline context the evolutionary algorithm is built on.","marker":"Yang et al. (2019)"}],"fun_headline_variants":["LLM acts as protein optimizer without any fine-tuning","Generic LLM proposes better protein variants in directed evolution","No fine-tuning: LLM improves protein fitness via evolution","LLM-driven evolution finds fitter protein sequences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim depends on the assumption that the LLM's proposals are driven by genuinely useful knowledge about what makes proteins fit, rather than by memorized training examples or by the fitness scores placed in the prompt.","fun_headline_variants_meta":{"raw":{"variants":["LLM acts as protein optimizer without any fine-tuning","Generic LLM proposes better protein variants in directed evolution","No fine-tuning: LLM improves protein fitness via evolution","LLM-driven evolution finds fitter protein sequences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3038,"prompt_tokens":823,"completion_tokens":2215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":2152}},"tokens_in":439,"tokens_out":2215,"duration_ms":16387,"temperature":1.0,"reasoning_tokens":2152,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:06:16.016583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LLM-guided loop on a protein family that was not present in the LLM's pretraining corpus (for example, a newly designed enzyme with no close public homologs) and compare against random-mutation evolution; if the advantage disappears, the effect is memorization rather than optimization. Alternatively, remove or shuffle the fitness scores in the prompt and check whether proposal quality degrades.","supporting_citations":[{"cited_title":"A combinatorially complete epistatic fitness landscape in an enzyme active site","cited_arxiv_id":null,"evidence_quote":"Supplies the TrpB combinatorially complete active-site fitness landscape used as an exact oracle benchmark."},{"cited_title":"Local fitness landscape of the green fluorescent protein","cited_arxiv_id":null,"evidence_quote":"Supplies the GFP fluorescence deep-mutational-scanning dataset that defines the ML-oracle task."},{"cited_title":"Deep diversification of an aav capsid protein by machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies the AAV capsid deep-mutational-scanning dataset used as the second ML-oracle task."},{"cited_title":"Tuned fitness landscapes for benchmarking model-guided protein design","cited_arxiv_id":null,"evidence_quote":"Defines the SLIP tuned fitness landscape used to construct the synthetic Syn-3bfo oracle."},{"cited_title":"Machine-learning-guided directed evolution for protein engineering","cited_arxiv_id":null,"evidence_quote":"Establishes the ML-guided directed evolution paradigm and the random-mutation baseline context the evolutionary algorithm is built on."}],"review_version":1}