{"id":"78d8c45d-43f0-4b3e-8177-0a51442a782b","arxiv_id":"2411.08286","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Protein structures are encoded as 400-bit hash codes with a graph neural network and contrastive learning, giving faster and more memory-efficient similarity search than real-valued vector methods.","lead":"POSH is a machine learning method that turns each protein structure into a short binary fingerprint, or hash code, so similar proteins can be found by comparing simple bit patterns instead of aligning 3D shapes. The paper reports up to six times less memory and four times faster search than existing vector based methods, with competitive accuracy on standard protein benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The state-of-the-art accuracy claim is not currently supported: Table 1 shows POSH Top-1 on SCOPe (0.7242) below GraSR (0.7282), and GraSR results are copied without error bars, so the claimed superiority may be an evaluation artifact.","rationale":"The load-bearing point is the trustworthiness of the empirical SOTA claim. The authors' own Table 1 contradicts the abstract on Top-1, and the only learning baseline they beat on SCOPe (GraSR) is imported from another paper rather than re-run. Without error bars, the AUROC gap (0.9906 vs 0.9823) and Top-5/Top-10 gaps cannot be distinguished from noise. The reader's weakest-assumption concern about graph expressivity is a plausible scientific risk, but it is less directly checkable and less immediate than the fact that the headline comparison is not currently reproducible. This is not a charge of misconduct; it is a request for standard empirical evidence. Because the efficiency advantage is inherent to binary codes and the architecture is coherent, conditional acceptance remains appropriate, with the controlled re-run as the condition.","tokens_in":11000,"tokens_out":10462,"duration_ms":122407,"concrete_test":"Re-run GraSR from its released code on the identical SCOPe and ind_PDB splits and evaluation protocol as POSH, and run POSH with at least five random seeds; report mean and standard deviation for every metric in Tables 1 and 2. If GraSR's copied numbers are not reproduced, or if POSH's margins are within one standard deviation, the state-of-the-art claim is not established; if the margins persist, the claim stands as conditional, ideally with Foldseek added to define scope.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: binary codes reduce time/memory, and POSH is state-of-the-art in accuracy. The efficiency part is structurally sound because Hamming search over short binary codes is cheap. The accuracy part is not yet trustworthy. In Table 1, POSH Top-1 on SCOPe is 0.7242 versus GraSR 0.7282; the text admits POSH trails on this metric. This is in tension with the Abstract's unqualified statement that POSH outperforms other methods. Additionally, Section 4.1 says GraSR results are directly copied from the original paper, while POSH is reported without error bars or multiple-seed variance. The AUROC/AUPRC/Top-5/Top-10 gaps could be within run-to-run noise, and the copied baseline comes from a separate evaluation pipeline whose exact splits and preprocessing are only assumed identical. Foldseek (Ref. 18) is cited but not benchmarked, leaving the scope of the SOTA claim unclear. This is an empirical evidence gap, not an internal inconsistency in the learning objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes POSH, a learned binary hashing method for protein structure similarity search (PSSS). POSH represents each protein as a k-nearest-neighbor graph over backbone atoms, extracts hand-crafted node and edge features, and learns a graph encoder with node and edge update layers. The encoder output is binarized via a hashing loss, and training uses an InfoNCE contrastive loss with a sampling strategy that includes a TM-score-guided substructure augmentation. At test time, proteins are encoded into binary codes and searched by Hamming distance with a length-scaling factor. Experiments on SCOPe, ind_PDB, and AlphaFold-scale databases claim state-of-the-art accuracy and substantial memory/time savings over alignment-free baselines.","tokens_in":11236,"tokens_out":5380,"duration_ms":54165,"significance":"If substantiated, the paper would make a useful contribution by introducing a hashing-based representation for PSSS that replaces real-valued vectors with binary codes, enabling more efficient large-scale search. The method is internally coherent, and the graph encoder with edge features is a reasonable design choice. The authors also contribute a substructure sampling strategy and a distance-scaling rule for Hamming search, which are sensible and likely transferable to other structure-based retrieval tasks. However, the central accuracy claim is not fully supported by the reported experiments: the paper claims state-of-the-art accuracy while acknowledging a Top-1 deficit on SCOPe, and the comparisons lack error bars and rerun baselines. The efficiency claims are plausible but need more careful benchmarking to rule out implementation artifacts.","major_comments":[{"comment":"The abstract and introduction claim that POSH can 'outperform other methods to achieve state-of-the-art accuracy,' but Table 1 shows POSH Top-1 on SCOPe is 0.7242 versus GraSR's 0.7282. Section 4.2 acknowledges this by saying POSH trails on Top-1 and is only 'comparable' there. This is a direct contradiction of an unqualified claim. The accuracy claim needs to be qualified to the specific metrics where POSH actually leads, or the abstract should be rephrased. As written, the central claim of state-of-the-art accuracy is not supported by the paper's own table.","section":"Abstract and Section 1 vs. Table 1"},{"comment":"The results for GraSR are 'directly copied from its original paper' while POSH is trained and evaluated under the authors' own pipeline. The paper states that the same dataset and filtering criteria are used, but it does not demonstrate that the evaluation protocol is identical, including the definition of similar pairs, the Top-k calculation, the database/query split, and the post-processing of rankings. Without rerunning GraSR under the exact same evaluation code, the comparison may systematically favor POSH. The authors should either rerun GraSR (and ideally the other learning-based baseline) using the same evaluation harness, or provide a detailed point-by-point protocol match and, if discrepancies exist, quantify their effect.","section":"Section 4.1, Baseline comparison with GraSR"},{"comment":"All accuracy tables report single point estimates with no error bars, confidence intervals, or number of seeds. This is a concern because the differences between POSH and GraSR are modest on several metrics (e.g., AUROC 0.9906 vs 0.9823, Top-5 0.7225 vs 0.7101 on SCOPe). Given stochastic mini-batch sampling, substructure sampling, and random initialization, these gaps could fall within run-to-run variance. The authors should report mean and standard deviation over at least three independent training runs, and ideally a paired significance test, before claiming consistent superiority.","section":"Tables 1, 2, 4, and 5, statistical reliability"},{"comment":"The time cost comparison lacks implementation details for the baselines. It is not stated whether all methods use exact linear scan, whether the baseline vector search is implemented with optimized BLAS routines or a straightforward loop, what hardware and software versions are used, or how the database vectors are stored and loaded. The reported speedup of 'more than four times' over GraSR may be sensitive to these implementation choices. The authors should specify the search algorithm and implementation environment for every method, and ideally release the benchmarking code so that the efficiency claim is reproducible.","section":"Section 4.2, Figure 4, time cost benchmark"},{"comment":"The paper cites Foldseek [18] but does not benchmark it or discuss it as a related approach. Foldseek also uses a discrete representation of protein structures (a 3Di alphabet) and is a state-of-the-art fast structure search tool. The claim that POSH is 'the first hashing method for PSSS' requires a clear distinction from Foldseek's discrete encoding, which could be viewed as a form of symbol-level hashing. At minimum, Foldseek should be included in the accuracy and efficiency comparisons, or the authors should explicitly argue why it is outside the scope of the claim. Without this, the novelty and SOTA positioning are unclear.","section":"Related Works and Section 1, 'first hashing method' claim"},{"comment":"The paper reports time and memory results for databases up to the size of the AlphaFold database (approximately 200 million proteins), but it does not explain how the binary codes or real-valued vectors for these databases are obtained. If the authors actually encoded 200 million structures, the computational cost of that step should be reported. If the numbers are extrapolated or simulated, the extrapolation method must be described. This is necessary to judge whether the claimed speedup and memory savings are measured or projected, and whether the comparison is fair.","section":"Section 4.2 and Table 3, AlphaFold database experiments"}],"minor_comments":[{"comment":"The kNN graph construction may produce a directed graph if the nearest-neighbor relation is not symmetric; the aggregation uses the neighbor set N_i without clarifying whether the graph is symmetrized. Please state whether the graph is made undirected.","section":"Section 3.2, Eq. (1)"},{"comment":"The term (yt1)^2 uses '1' without definition. Please define 1 as the all-ones vector of appropriate dimension.","section":"Section 3.3, Eq. (3)"},{"comment":"The ablation study does not state which dataset is used (SCOPe or ind_PDB). Please specify the dataset in the table caption or text.","section":"Section 4.3, Table 4"},{"comment":"The text says 'with the growing code length, the accuracy of the models consistently improves,' but POSH-400* has AUROC 0.9708, which is lower than POSH-256's 0.9720. This is an internal inconsistency; either the statement should be qualified or the explanation for the non-monotonic AUROC should be given.","section":"Appendix B.2, Table 5"},{"comment":"The procedure for precomputing the minimum sampling length per structure to satisfy TM(P, Ps) >= alpha is not described. Please provide the algorithm or a precise description of how this length is found.","section":"Section 3.4, substructure sampling"},{"comment":"Several references are incomplete: [18] lacks volume and page information, and [20] is cited as an arXiv preprint although a published version exists. Please update all references to their final published versions.","section":"References"},{"comment":"The architecture diagram is dense and does not label the node update, edge update, hashing layer, and contrastive loss components in the figure itself. Adding direct labels would improve readability.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core idea (learned binary hashing for protein structure search) is interesting and the method appears technically sound, but the manuscript's headline accuracy claim is not supported by its own Table 1, and the empirical comparison needs to be strengthened with rerun baselines, error bars, and more careful efficiency benchmarking. The 'first hashing method' claim also needs to be positioned against Foldseek. These are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Han and Li's POSH paper is worth reading if you care about protein structure search at scale. The core result is believable: representing structures as 400-bit binary codes and searching by Hamming distance gives a 4x speedup and a 6x memory saving over the standard real-valued baselines. That part is not in dispute—the arithmetic of binary codes versus floats makes it structurally plausible, and the experiments line up.\n\nWhat is actually new: POSH is, as far as I know, the first learned hashing method for PSSS. The specific recipe—kNN graph on Cα distances, backbone bond/dihedral features on nodes, RBF-encoded inter-backbone-atom distances on edges, an L-layer message-passing encoder, InfoNCE plus a hashing loss—is a coherent engineering package. The substructure sampling trick with a TM-score guard at α=0.9 is a sensible way to generate diverse positive pairs without wrecking similarity. The authors also show ablations for edge features, substructure sampling, and distance scaling, which is more than many papers at this level do.\n\nWhere it gets soft: the abstract says POSH 'outperforms other methods to achieve state-of-the-art accuracy,' but on SCOPe Top-1 it trails GraSR (0.7242 vs 0.7282). The text admits this in Section 4.2, so it's an overstatement in the abstract rather than a hidden result. More importantly, there are no error bars or multiple-seed runs anywhere, and the GraSR numbers are copied from the original paper rather than recomputed. That makes it hard to know whether the AUROC/AUPRC gains and the Top-1 deficit are real or within run-to-run noise. Foldseek is cited but never benchmarked, so the SOTA claim is implicitly scoped to real-valued alignment-free methods; that scope should be stated. The time benchmark (Figure 4) also lacks enough detail to know whether query encoding, sorting, or I/O is included. None of these are fatal—the efficiency result is structurally sound—but they are the difference between a claim I can lean on and one I have to verify.\n\nThe limitation section is honest but narrow; it doesn't address the fact that training requires pairwise TM-scores, which cap scalability, nor the missing baseline comparisons.\n\nWho should read this: anyone building a large-scale structure search pipeline, and people working on learned representations for proteins. It deserves a serious referee. I'd send it to review, and ask for code, variance estimates, a rerun or explicit justification of GraSR numbers, and either a Foldseek comparison or a clearly scoped SOTA claim.","headline":"A genuinely useful hashing method for protein search, but the SOTA accuracy claim is one metric short and the empirical reporting needs tightening.","tokens_in":11781,"tokens_out":2228,"would_cite":true,"duration_ms":23997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that protein structures can be hashed into binary codes that make similarity search over six times more memory-efficient and over four times faster than existing vector methods, while achieving state-of-the-art accuracy.","keywords":["protein structure similarity search","hashing","binary hash codes","graph neural network","contrastive learning","TM-score","Hamming distance","alignment-free representation"],"falsifier":"Build a test set of protein pairs with nearly identical C-alpha traces and backbone angles but different side-chain packing or rewired nonlocal contacts; if POSH's Hamming rankings rate these pairs as similar while their TM-scores are low, the graph features rather than the hashing objective are the bottleneck, and the claim that the full method preserves structural similarity fails.","tokens_in":10749,"feed_emoji":"🧬","tokens_out":6945,"duration_ms":67366,"temperature":0.7,"pith_summary":"The paper proposes POSH, which it calls the first hashing method for protein structure similarity search. The central claim is that a learned binary hash code of fixed length can stand in for a protein structure well enough that searching a database by Hamming distance matches or beats the accuracy of existing alignment-free real-valued vector methods, while using over six times less memory and running over four times faster. This matters because alignment-based search is impractical at the scale of modern structure databases, and even vector-based alignment-free methods need hundreds of gigabytes to index hundreds of millions of predicted structures. POSH trains a graph encoder with hand-crafted backbone features and contrastive learning, with TM-score as the ground-truth similarity signal, so the claim is that a 400-bit code preserves enough structural information for retrieval.","feed_headline":"Hashing folds protein search into binary codes, cutting costs by 6x","feed_subtitle":"Learned 400-bit protein codes match or beat top accuracy while making billion-scale structure lookup practical.","key_machinery":"The central object is the protein structure graph: nodes are amino acids, and edges are the `$k$` nearest neighbors by `$C_\\alpha$` distance. The argument is carried by three components working together: hand-crafted node features (sine and cosine of backbone bond angles and dihedral angles) and edge features (Gaussian RBF encodings of interatomic distances among C, `$C_\\alpha$`, N, O, and `$C_\\beta$` atoms); a structure encoder that alternates node update and edge update message-passing layers, so that both node and edge interactions are modeled; and a hashing-plus-contrastive objective that trains the pooled node representation toward a binary sign code while making Hamming distances track TM-score similarity. A substructure sampling strategy guided by a TM-score floor of 0.9 diversifies positive examples, and a length-scaling term breaks ties among equal Hamming distances. The name of the method, POSH, stands for protein structure hashing.","core_discovery":"On its own terms, the paper establishes that protein structure similarity can be compressed into short binary codes without sacrificing retrieval quality. POSH builds a graph per protein whose nodes are amino acids and whose edges connect each residue to its `$k$` nearest neighbors by `$C_\\alpha$` distance; raw node features are sine and cosine encodings of backbone bond and dihedral angles, and raw edge features are RBF-encoded distances between backbone and `$C_\\beta$` atoms. A graph neural network with alternating node and edge message-passing layers produces node representations, which are pooled and linearly mapped to a real-valued code; a hashing loss pulls this code toward its sign-binarized form, and an InfoNCE contrastive loss pushes hash codes of TM-score-similar proteins together and dissimilar ones apart. With the resulting 400-bit codes, POSH reports accuracy above all compared baselines on SCOPe (AUROC 0.9906, AUPRC 0.6853) and on the independent ind_PDB set (AUROC 0.9699, AUPRC 0.4719), with a memory compression ratio of roughly 1955 times relative to storing raw structures, and search time more than four times faster than the best real-valued baseline.","pith_inferences":["A testable extension the paper leaves open is whether the same binary-code pipeline can be trained from predicted pairwise similarity scores that are cheaper than full alignment, since the current training set-up inherits the cost of alignment-based labels.","The graph representation uses only backbone and C-beta atoms; an inference is that adding side-chain contact maps or long-range residue-residue contacts could push accuracy higher on multi-domain proteins, where the paper itself reports a drop relative to single-domain SCOPe.","If the contrastive hash codes really track TM-score, they could also serve as a filtering pre-step before alignment: a shortlist of Hamming-near proteins could be re-ranked by the alignment procedure, reducing the costly alignment step to candidate proteins only.","The reported compression ratio of 1955 times suggests that far shorter codes may suffice for clustering large structure databases; the paper's code-length experiments only go down to 64 bits."],"forward_implications":["If the central claim holds, structure databases the size of the AlphaFold database can be indexed in about 11 GB of binary codes instead of hundreds of gigabytes, putting billion-scale search on a single machine.","A model trained on single-domain SCOPe structures transfers to the multi-domain ind_PDB set with top accuracy, indicating the learned hash codes capture reusable structural features rather than dataset-specific artifacts.","Code length controls a clean accuracy-cost trade-off: accuracy rises with code length from 64 to 512 bits, so deployment can pick the shortest code that meets a target recall.","Because Hamming distance ties are broken by protein length, retrieval rankings in the top-k regime improve, which is the regime users actually see.","The ablation results imply that edge features, substructure sampling, and distance scaling all contribute independently, so the reported gains are not coming from hashing alone."],"supporting_citations":[{"why":"Supplies the alignment-based pairwise similarities (TM-score) used to define positive and negative training samples.","marker":"[1]"},{"why":"Defines the TM-score metric itself, which is the paper's ground-truth notion of structural similarity for training and evaluation.","marker":"[16]"},{"why":"Provides the contrastive graph neural network baseline and the dataset filtering criteria that POSH adopts for fair comparison.","marker":"[15]"},{"why":"Provides the learning-based baseline on which code-length and experimental settings are modeled.","marker":"[14]"},{"why":"Provides the InfoNCE contrastive loss that drives hash codes of similar proteins together and dissimilar proteins apart.","marker":"[29]"},{"why":"Provides the deep supervised hashing loss that encourages the real-valued code to approach its binary sign code.","marker":"[21]"},{"why":"Supplies the virtual C-beta atom construction used when a residue's C-beta atom is missing in the edge features.","marker":"[26]"},{"why":"Supports the edge-update design and contributes the substructure sampling idea that POSH adapts with a TM-score constraint.","marker":"[28]"},{"why":"Supports the claim that neglecting edge updates is suboptimal, justifying the edge update layer.","marker":"[27]"}],"fun_headline_variants":["Protein search shrinks to 400-bit hashes, 4x faster, 6x lighter","Binary codes slash protein search memory 6x and speed 4x","Hashing protein structures keeps accuracy while cutting costs","POSH: 400-bit protein codes match top accuracy at 6x lower memory","Protein similarity search made practical with binary hashing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the k-nearest-neighbor graph built from C-alpha distances plus backbone bond and dihedral angles contains enough information for the graph encoder to learn hash codes that reproduce TM-score similarity; if side chains, nonlocal contacts, or domain organization are essential to structural similarity, the hand-crafted graph caps accuracy regardless of the hashing layer.","fun_headline_variants_meta":{"raw":{"variants":["Protein search shrinks to 400-bit hashes, 4x faster, 6x lighter","Binary codes slash protein search memory 6x and speed 4x","Hashing protein structures keeps accuracy while cutting costs","POSH: 400-bit protein codes match top accuracy at 6x lower memory","Protein similarity search made practical with binary hashing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1571,"prompt_tokens":1052,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":668,"tokens_out":519,"duration_ms":5511,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:46:02.859252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of protein pairs with nearly identical C-alpha traces and backbone angles but different side-chain packing or rewired nonlocal contacts; if POSH's Hamming rankings rate these pairs as similar while their TM-scores are low, the graph features rather than the hashing objective are the bottleneck, and the claim that the full method preserves structural similarity fails.","supporting_citations":[{"cited_title":"Tm-align: a protein structure alignment algorithm based on the tm-score","cited_arxiv_id":null,"evidence_quote":"Supplies the alignment-based pairwise similarities (TM-score) used to define positive and negative training samples."},{"cited_title":"Fast protein structure comparison through effective representation learning with contrastive graph neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive graph neural network baseline and the dataset filtering criteria that POSH adopts for fair comparison."},{"cited_title":"Learning structural motif representations for efficient protein structure search","cited_arxiv_id":null,"evidence_quote":"Provides the learning-based baseline on which code-length and experimental settings are modeled."},{"cited_title":"Feature learning based deep supervised hashing with pairwise labels","cited_arxiv_id":null,"evidence_quote":"Provides the deep supervised hashing loss that encourages the real-valued code to approach its binary sign code."},{"cited_title":"Lozano, Payel Das, and Jian Tang","cited_arxiv_id":null,"evidence_quote":"Supports the edge-update design and contributes the substructure sampling idea that POSH adapts with a TM-score constraint."}],"review_version":1}