Pith. sign in

REVIEW 5 major objections 5 minor 13 references

Protein Structure Prediction in the 3D HP Model Using Deep Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep Q-learning with reservoir and attention networks matches or approaches best-known 3D HP folding energies while using fewer training episodes.

desk verdict A competent empirical application of standard DQN variants to the 3D HP model, undone by an abstract that overstates its own Table 3. read the letter →

arxiv 2412.20329 v1 pith:IV5RLNIV submitted 2024-12-29 cs.LG cs.AIq-bio.BM

Giovanny Espitia , Yui Tik Pang , James C. Gumbart This is my paper · ORCID
classification cs.LGcs.AIq-bio.BM
keywords 3DHPmodelproteinstructurepredictiondeepreinforcementlearningreservoircomputingLSTMmulti-headattentionQ-learningself-avoidingwalk
topics Protein Folding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that deep Q-learning, equipped with architectures chosen for temporal memory, can solve instances of the NP-complete 3D Hydrophobic-Polar (HP) protein folding problem as well as the best known specialized solvers while training more efficiently. For chains up to 36 residues, a reservoir-based feedforward network (FFNN-R) is claimed to reach the best-known energy values with about 25% fewer training episodes than a plain fully connected network. For longer chains, an LSTM with multi-head attention (LSTM-A) is claimed to match or closely approach best-known energies. If the claims hold, deep reinforcement learning becomes a practical, sample-efficient alternative to genetic algorithms and Monte Carlo methods on this benchmark.

What carries the argument

The load-bearing objects are the two Q-network architectures. FFNN-R applies a fixed random reservoir update $r(t) = f(W_{\mathrm{in}} x(t) + W r(t-1))$ with a tanh activation and sparse random connectivity, projecting the 8-dimensional one-hot state into a high-dimensional temporal representation that is then passed through fully connected layers. LSTM-A runs the state sequence through an LSTM of hidden size 512 and pools the hidden states with four-head attention, $\mathrm{MultiHead}(H) = \mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_4)W_O$, before a final layer outputs per-action Q-values. Both are trained as DQNs with experience replay, a target network, and a smooth L1 loss, so the claimed gains come from the state representation and architecture rather than new reinforcement-learning machinery.

What would settle it

Train FFNN-R and LSTM-A on held-out HP sequences not in the published benchmark, using the same one-hot state and hyperparameters, and compare final energies to exhaustive or best-known values. A clear failure on held-out sequences, or a measurable change in performance when the state is augmented with an explicit occupancy map, would show that the claimed optimality depends on implicit path memory rather than on a general folding policy.

Watch

Extended reading notes

Core claim

The central claim is that the two architectures make Deep Q-Networks competitive with dedicated HP-model solvers: FFNN-R, a fully connected network fed by a fixed random reservoir, reaches best-known energies on most short benchmark sequences (A1–A5 and A8–A10) with roughly 25% fewer episodes than the vanilla FFNN baseline, and LSTM-A, an LSTM with multi-head attention, matches best-known energies on the 3d1–3d4 sequences and comes within a few hydrophobic contacts of best-known values on 3d5–3d9. The paper takes this as evidence that implicit temporal memory (reservoir) and long-range attention (LSTM-A) are the ingredients that let DQN scale on the 3D HP model.

Load-bearing premise

The load-bearing premise is that the 8-dimensional one-hot state, which records only the available actions and the current bead's H/P type, conveys enough information for the network to reconstruct the self-avoiding path from the history of actions; if this implicit memory is ambiguous, the reported optimality is tied to the specific benchmark sequences and will not transfer to new HP instances.

Editorial extensions

If this is right

  • For proteins under 36 residues, FFNN-R reaches best-known 3D HP energies with about 25% fewer training episodes than a vanilla fully connected DQN, making deep RL a faster route to optimal short-chain folds.
  • For chains of 46–60 residues, LSTM-A outperforms the plain FFNN and the LSTM that uses only its last hidden state, suggesting multi-head attention is what lets DQN capture long-range interactions.
  • The attention weights start specializing only after about 100,000 episodes, which implies the model first learns local folding patterns and then uses attention for global assembly.
  • Both architectures use the same stabilized DQN framework, so the efficiency difference is attributable to the two architectural changes, not to replay or target-network tricks.
  • Across the standard benchmark, reported final energies match best-known values on the short sequences and approach them on the longest ones, establishing a new sample-efficiency point for DRL-based HP folding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 25% episode reduction is measured against the paper's own vanilla FFNN, not against genetic algorithms, PERM, or other solvers, so the sample-efficiency claim should be read as architecture-to-architecture rather than method-to-method.
  • Because the state encoder omits lattice coordinates and the occupancy map, the learned policy may be capturing sequence-specific action statistics rather than general 3D geometry; adding an explicit occupancy-map channel is a natural test of this.
  • The observed specialization of attention heads into periodic and position-specific patterns hints that LSTM-A is learning secondary-structure-like motifs and long-range contacts separately, which could be exploited to seed folding on even longer chains.
  • A hybrid that lets FFNN-R propose short local fragments and LSTM-A assemble them could sidestep FFNN-R's degradation beyond length 36 and LSTM-A's quadratic memory cost, though the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes two deep reinforcement learning architectures for protein structure prediction in the 3D Hydrophobic-Polar (HP) lattice model: a reservoir-based feedforward network (FFNN-R) for short sequences (N <= 36) and an LSTM with multi-head attention (LSTM-A) for longer sequences. The authors formulate protein folding as a self-avoiding walk and frame it as a Markov decision process solved with stabilized Deep Q-Learning. They report energy values on the 3d and A benchmark sets, claim that both architectures consistently match best-known energy values (BKV), and claim that FFNN-R reaches optimal conformations with approximately 25% fewer training episodes than the vanilla FFNN. The paper also discusses attention patterns, parameter counts, and training times, and releases code via GitHub.

Significance. If the central claims were correct, this would be a noteworthy contribution: it would introduce reservoir computing to 3D HP protein folding, demonstrate an attention-enhanced LSTM that matches BKVs on long benchmark sequences, and provide an unusually sample-efficient DQN variant, with code released for reproducibility. However, the headline claim of consistent optimality is directly contradicted by the paper's own Table 3, and the efficiency claim is not supported by the logged experimental data in Table A.1. The architectural ideas are interesting, but the reported results do not establish the stated contribution.

major comments (5)
  1. [Abstract and Section 1 vs. Table 3] The abstract's claim that 'both architectures consistently achieve optimal conformations matching the best known energy values' is contradicted by Table 3. For the long sequences, LSTM-A reports -33, -30, -32, -40, and -51 for 3d5, 3d6, 3d7, 3d8, and 3d9, respectively, versus BKVs of -35, -31, -34, -44, and -55. For the short A-set, FFNN-R reports -11 and -12 for A6 and A7 versus BKVs of -12 and -13. By the paper's own standard, 7 of the 19 benchmark entries are not optimal, so the 'consistently' claim is internally inconsistent with the reported data.
  2. [Table A.1 vs. Table 3] The right-most column of Table 3, described as 'the least energy conformation found by our models,' is inconsistent with Table A.1. For A4, Table A.1 lists FNN-RESERVOIR's Best E_min as -14 while Table 3 reports -15; for A9, Table A.1 lists Best E_min as -6 while Table 3 reports -7. This indicates that the reported column mixes different statistics (E_Min versus Best E_min) across rows, so the reader cannot determine what quantity is actually being reported.
  3. [Section 3.1, Efficiency] The claim that FFNN-R converges in about 200,000 episodes and requires 'approximately 25% fewer training episodes' is unsupported by the logged data. Table A.1 records 200,000 training episodes for both FNN-VANILLA and FNN-RESERVOIR on A1-A10, and it gives no information about the episode at which each run first reached its best energy. Without first-convergence episode counts, the 25% reduction claim has no evidentiary basis.
  4. [Section 4, Discussion] The discussion states that 'both FFNN and FFNN-R demonstrate robust performance across multiple independent training runs' and that the architectures 'show consistent performance with lower variance in final energy values,' but no variance statistics, standard deviations, or repeated-run results are reported anywhere in the paper. The reported energies appear to be best-of-run values, so the consistency claim is not established.
  5. [Section 2.2.3, State Representation] The 8-dimensional one-hot state encodes only six action indicators and the H/P type of the current bead; it does not directly expose lattice coordinates or an occupancy map. The paper asserts that the network 'deduces this information from actions taken thus far,' but no experiment verifies that the implicit action-history memory is sufficient to reconstruct the self-avoiding path. Without such a verification, the claim that the architectures learn general folding behavior cannot be separated from memorization of the specific benchmark sequences.
minor comments (5)
  1. [Title] The title contains a formatting error: 'M ODEL' should be 'MODEL'.
  2. [Table A.1] The header 'FNN-V ANILLA' contains a typo and should read 'FNN-VANILLA'.
  3. [Table A.1] Several rows in Table A.1 are missing parameter counts or training times (e.g., LSTM-OLH 3d2, FNN-VANILLA rows), making the efficiency comparison incomplete.
  4. [Section 2.3.1] The reservoir input weight matrix is described as 'trainable' in Eq. (12), while the text says the reservoir weight matrix W is 'randomly initialized and remains fixed'; this apparent inconsistency in terminology should be clarified.
  5. [Section 4.1] The statement in the Limitations section that 'our implementation achieves state-of-the-art results' is not supported by Table 3, which shows several entries below the quoted BKVs and below the GA baseline for long sequences.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is an empirical benchmark against external best-known energy values, with the reward function defined independently of the architectures.

full rationale

The paper's derivation chain is an empirical reinforcement-learning benchmark, not a closed-form derivation. The reward is the standard HP-model energy E = -(number of valid H-H contacts), which is defined physically and independently of the proposed architectures. The target values are external best-known energies (BKV) taken from Boumedine and Bouroubi (2022), and the authors' models are compared against those external numbers. There is no fitted parameter that is later renamed as a prediction, no equation that reduces to its own input, and no load-bearing self-citation chain: the cited prior work (e.g., Yang et al., Mnih et al., Vaswani et al.) is used for standard modeling conventions and algorithmic components, not to establish the paper's empirical conclusions. The paper's internal inconsistency between its abstract's claim of 'consistently achieve optimal conformations' and Table 3's reported sub-BKV energies on several sequences is a correctness and reporting concern, not a circularity concern. Likewise, the unimplemented state-representation concern (the network does not directly observe lattice coordinates) is a validity or Markov-assumption concern, not an identity between input and output. Because the central claims are evaluated against external benchmarks and the reward function is not defined in terms of the claimed results, no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

No new physical entities or physically meaningful fitted constants are introduced. The only fitted quantities are network hyperparameters and episode counts chosen per benchmark; the load-bearing assumptions are the validity of the HP model, the correctness of the quoted BKVs, and the sufficiency of the one-hot state representation for the folding MDP.

free parameters (7)
  • Reservoir size N_r = 1000 for N<=36; 3000 for longer
    Chosen empirically in Section 2.3.1 based on sequence length; FFNN-R performance and memory footprint depend on it.
  • Attention head count = 4
    Section 4 states 4 heads 'proves optimal' but no systematic ablation or search is reported.
  • Hidden dimension = 512
    Used for LSTM hidden state and fully connected layers in both architectures; no justification beyond typical practice.
  • Batch size = 16 or 32
    Selected per sequence in Table A.1; Section 4 says batch 32 is optimal but no comparison is shown.
  • Number of training episodes = 200K to 750K per sequence
    Each run in Table A.1 uses a hand-set episode count; support for the '25% fewer episodes' claim is absent.
  • Learning rate = 0.001
    Adam optimizer learning rate chosen in Section 3; a hand-set hyperparameter that affects convergence.
  • Undisclosed DQN hyperparameters = not stated
    Epsilon schedule, discount factor gamma, replay buffer size, target network update period, and reward shaping are never given, yet the learning dynamics depend on them.
assumptions (4)
  • domain assumption The 3D HP model with energy E = -(number of H-H contacts) is a meaningful target for protein structure prediction.
    Invoked throughout Section 2.1; the entire benchmark evaluation rests on this coarse model being a valid proxy.
  • domain assumption The best-known energy values (BKV) quoted from Boumedine and Bouroubi (2022) are the true optimal energies.
    Used as ground truth in Section 3 and Table 3; if a BKV is not optimal, 'matching BKV' does not establish optimality.
  • domain assumption The one-hot state with six action indicators and H/P type, plus reservoir or LSTM memory, is a sufficient and Markovian representation for the folding MDP.
    Section 2.2.3 says the network deduces placements from actions taken so far, but no verification is provided that this representation disambiguates all self-avoiding paths.
  • standard math Standard DQN convergence assumptions, including experience replay, target networks, and epsilon-greedy exploration, apply to this sparse-reward episodic task.
    Section 2.2.2 adopts the DQN framework of Mnih et al. 2015; no analysis shows that the sparse terminal reward is learnable by these networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Protein Structure Prediction in the 3D HP Model Using Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/IV5RLNIV

@misc{pith2026241220329,
  author       = {Pith},
  title        = {Pith review of: Protein Structure Prediction in the 3D HP Model Using Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IV5RLNIV}},
  note         = {Machine review of arXiv:2412.20329}
}
read the original abstract

We address protein structure prediction in the 3D Hydrophobic-Polar lattice model through two novel deep learning architectures. For proteins under 36 residues, our hybrid reservoir-based model combines fixed random projections with trainable deep layers, achieving optimal conformations with 25% fewer training episodes. For longer sequences, we employ a long short-term memory network with multi-headed attention, matching best-known energy values. Both architectures leverage a stabilized Deep Q-Learning framework with experience replay and target networks, demonstrating consistent achievement of optimal conformations while significantly improving training efficiency compared to existing methods.

Figures

Figures reproduced from arXiv: 2412.20329 by the authors.

Figure 1
Figure 1. Deep reinforcement learning training loop. In a), we sample a batch of experience from the buffer. The batch [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The input layer consists of a (N, 8, 1) tensor representing the state at a particular timestep. The reservoir [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The multi-head attention mechanism enhances the network’s ability to focus on different aspects of the input sequence simultaneously. Given an input sequence processed by the LSTM layers producing hidden states H ∈ R N×d , where N is the sequence length and d is the hidden dimension, the attention mechanism computes attention patterns across four different representation subspaces. For each attention head i, the mec… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: LSTM-A architecture for protein folding. Sequential states are processed through LSTM cells, generating [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 4
Figure 4. Figure 4: Least Energy Conformations for different sequences. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Plots a) 3d1 and b) 3d5 show the minimum conformation energy as a function of episode. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Plots a) A2 and b) A4 show the minimum conformation energy as a function of episode. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Evolution of multi-head attention weights during protein structure prediction training. Each row shows the [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

  1. [3]

    T. C. Beutler and K. A. Dill. A fast conformational search strategy for finding low energy structures of model proteins. Protein Sci., 5(10):2037–2043, 10

    work page

  2. [6]

    doi: 10.1016/j.asoc.2014.09.047. P. Grassberger. Pruned-enriched Rosenbluth method: Simulations of θ polymers of chain length up to 1 000

    work page doi:10.1016/j.asoc.2014.09.047 2014

  3. [9]

    doi: 10.1038/nature14236. A. L. Patton, W. F. Punch, and E. D. Goodman. A standard GA approach to native protein conformation prediction. Int. Conf. Genet. Algorithms, pages 574–581, 7

    doi:10.1038/nature14236

  4. [1993]

    doi: 10.1006/jmbi.1993.1258. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. arXiv, 30:5998–6008, 6

    work page arXiv 1993

  5. [1996]

    doi: 10.1002/pro.5560051010. N. Boumedine and S. Bouroubi. Protein folding in 3D lattice HP model using a combining cuckoo search with the Hill-Climbing algorithms. Appl. Soft Comput., 119:108564, 4

    work page doi:10.1002/pro.5560051010

  6. [1997]

    doi: 10.1103/physreve.56.3682. H. Jaeger. Echo state network. Scholarpedia, 2(9):2330, 1

    doi:10.1103/physreve.56.3682

  7. [2005]

    doi: 10.1186/1471-2105-6-30. A. Subramoney, F. Scherr, and W. Maass.Reservoirs learn to learn . 1

    work page doi:10.1186/1471-2105-6-30

  8. [2007]

    doi: 10.4249/scholarpedia.2330. N. Krasnogor. Studies on the theory and design space of memetic algorithms. PhD Thesis, 6

    work page doi:10.4249/scholarpedia.2330

Show all 13 references
  1. [2015]

    doi: 10.1109/tnnls.2014.2323247. 11 F. A. Araujo, M. Riou, J. Torrejon, S. Tsunegi, D. Querlioz, K. Yakushiji, A. Fukushima, H. Kubota, S. Yuasa, M. D. Stiles, and J. Grollier. Role of non-linear data processing on speech recognition task in the framework of reservoir computin...

    2014

  2. [2020]

    doi: 10.1038/s41598-019-56991-x. D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and translate.arXiv, 1

    doi

  3. [2021]

    doi: 10.1007/978-981-13-1687-6_3. R. Unger and J. Moult. Genetic algorithms for protein folding simulations. J. Mol. Biol., 231(1):75–81, 5

    doi

  4. [2022]

    doi: 10.1016/j.asoc.2022.108564. G. Czibula, M.-I. Bocicor, and I.-G. Czibula. Solving the protein folding problem using a distributed Q-Learning approach. Technical Report 3, Babe¸ s-Bolyai Univ.,

    2022

  5. [2023]

    A Experiments All of the conducted experiments are tabulated in Table A.1

    doi: 10.1016/j.physa.2022.128395. A Experiments All of the conducted experiments are tabulated in Table A.1. 12 Table A.1: 3D Trial Runs, Dictionary: LSTM-OLH → LSTM - OnlyLastHidden, LSTM-A (Num of Heads) → LSTM - with Attention, FNN-V ANILLA→ Without Reservoir, FNN-Reservoir...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.