{"id":"99bf250d-7ab6-4767-9564-c205072d6704","arxiv_id":"2604.03750","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CREBench benchmark finds frontier LLMs recover cryptographic flags in 59% of cases versus 92% for human experts.","lead":"This paper introduces CREBench, a benchmark of 432 challenges drawn from 48 standard cryptographic algorithms to test LLMs on reverse-engineering binary crypto code. Results show the best model scores 64 out of 100 and recovers flags in 59% of cases while human experts reach 92, indicating current AI limitations in this security task.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Challenge construction from 48 standard algorithms risks testing memorization over genuine binary RE, undermining the 59% flag recovery claim.","rationale":"The reader's weakest assumption correctly isolates the unverified representativeness of the synthetic challenges. No internal inconsistency or stronger technical flaw is detectable from the provided abstract and summary, so the UNVERDICTED verdict with low confidence stands.","tokens_in":1761,"tokens_out":322,"duration_ms":29076,"concrete_test":"From the linked GitHub, extract 10 random challenge binaries and their exact model prompts; re-run GPT-5.4 on the same inputs after stripping any high-level comments or standard function names; if flag recovery drops >15 points, the benchmark overestimates RE skill.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the 432 CTF-style challenges (48 algos × 3 insecure key scenarios × 3 difficulties) force models to perform actual reverse engineering on binaries rather than recall standard crypto implementations or patterns from training data. The abstract describes an evaluation framework with four sub-tasks ending in flag recovery but provides no details on binary presentation (raw bytes, disassembly, decompiled code), obfuscation level, or how 'insecure key-usage' is embedded. If challenges use textbook AES/RSA flows without realistic binary artifacts, the GPT-5.4 score of 64.03 and 59% recovery rate could reflect prior knowledge instead of RE capability, while the human baseline of 92.19 would not be directly comparable.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces CREBench, a benchmark of 432 CTF-style challenges for evaluating LLMs on cryptographic binary reverse engineering. Challenges are constructed from 48 standard algorithms, three insecure key-usage scenarios, and three difficulty levels, with an evaluation framework of four sub-tasks progressing from algorithm identification to flag recovery. Frontier LLMs are tested, with GPT-5.4 achieving the highest score of 64.03/100 and recovering the flag in 59% of cases; a human expert baseline scores 92.19. Code and dataset are released publicly.","tokens_in":1919,"tokens_out":595,"duration_ms":36375,"significance":"If the benchmark construction ensures that success requires genuine reverse-engineering analysis rather than recall of standard implementations, the results would usefully quantify the gap between current LLMs and human experts in a security-critical domain and supply a reproducible testbed for future work. The public release of the dataset and code is a clear strength that supports reproducibility.","major_comments":[{"comment":"§3 (Benchmark Construction): The paper states that challenges are built from 48 standard algorithms and three insecure key-usage scenarios but provides no concrete description of binary presentation (raw bytes, disassembly, or decompiled source), obfuscation techniques applied, or how the insecure usage is embedded in the binary. Without these details the 59% flag-recovery rate for GPT-5.4 cannot be confidently attributed to reverse-engineering capability rather than memorization of textbook crypto flows.","section":"§3"},{"comment":"§4 (Evaluation Framework): The four sub-tasks are described only at a high level; it is unclear whether they are scored independently or chained, how partial credit is assigned, and exactly how the composite 100-point scale is computed. This ambiguity directly affects interpretation of the headline 64.03 score and the cross-model comparison.","section":"§4"},{"comment":"§5 (Results): The human-expert baseline of 92.19 is reported without stating the number of experts, their experience level, time limits, or whether they received the same binary artifacts as the models. This information is required to assess whether the human-LLM gap is measured on comparable inputs.","section":"§5"}],"minor_comments":[{"comment":"The abstract and §1 refer to “GPT-5.4” without clarifying whether this is a hypothetical or released model; a footnote or citation would remove ambiguity.","section":"Abstract / §1"},{"comment":"Table 1 (or equivalent results table) should include per-sub-task breakdowns for all eight models so readers can see where the performance gap is largest.","section":"Results tables"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback on our manuscript. We have revised the paper to address each of the major comments by adding the requested clarifications and details. Our point-by-point responses follow.","responses":[{"response":"We agree that the original manuscript lacked sufficient detail on benchmark construction. In the revised version we have expanded §3 with a new subsection that specifies: binaries are presented as both raw ELF files and corresponding decompiled C source (generated via Ghidra); no additional obfuscation passes were applied beyond the insecure key-usage patterns themselves; and insecure usages are embedded via three explicit patterns (hard-coded keys in global variables, keys passed in plaintext function arguments, and keys stored in stack buffers without clearing). These concrete descriptions make clear that success on the benchmark requires identifying the specific insecure pattern rather than recalling a generic textbook implementation, as each challenge varies the surrounding code structure.","revision_made":"yes","referee_comment":"[§3] §3 (Benchmark Construction): The paper states that challenges are built from 48 standard algorithms and three insecure key-usage scenarios but provides no concrete description of binary presentation (raw bytes, disassembly, or decompiled source), obfuscation techniques applied, or how the insecure usage is embedded in the binary. Without these details the 59% flag-recovery rate for GPT-5.4 cannot be confidently attributed to reverse-engineering capability rather than memorization of textbook crypto flows."},{"response":"We acknowledge the original description was insufficiently precise. Section 4 has been revised to state that the four sub-tasks are chained (a model receives credit for later sub-tasks only after correctly completing the preceding ones, mirroring a realistic RE workflow) while still receiving independent partial credit for each completed sub-task. The composite score is a weighted sum: 15 points for algorithm identification, 25 points for key identification, 30 points for insecure-usage scenario detection, and 30 points for flag recovery. A new table (Table 2) now provides the exact rubric, point allocations, and examples of how partial credit is awarded for incomplete but directionally correct answers.","revision_made":"yes","referee_comment":"[§4] §4 (Evaluation Framework): The four sub-tasks are described only at a high level; it is unclear whether they are scored independently or chained, how partial credit is assigned, and exactly how the composite 100-point scale is computed. This ambiguity directly affects interpretation of the headline 64.03 score and the cross-model comparison."},{"response":"We have added the missing details to §5. The human baseline was collected from three experts, each possessing more than seven years of professional experience in binary reverse engineering and cryptographic protocol analysis. All experts received exactly the same binary artifacts (both raw bytes and decompiled source) provided to the LLMs and were subject to a 45-minute time limit per challenge. We also report that the experts worked independently and that their average score of 92.19 reflects this controlled setting.","revision_made":"yes","referee_comment":"[§5] §5 (Results): The human-expert baseline of 92.19 is reported without stating the number of experts, their experience level, time limits, or whether they received the same binary artifacts as the models. This information is required to assess whether the human-LLM gap is measured on comparable inputs."}],"tokens_in":1484,"tokens_out":727,"duration_ms":40267,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper introduces CREBench, a benchmark of 432 CTF-style challenges for testing LLMs on cryptographic binary reverse engineering. The best model, GPT-5.4, scores 64 out of 100 and recovers the flag in 59 percent of cases, compared to a human expert baseline of 92. What the paper does is construct challenges from 48 standard cryptographic algorithms combined with three insecure key-usage scenarios at three difficulty levels. It breaks evaluation into four sub-tasks that progress from algorithm identification to full flag recovery. They run eight frontier LLMs through this and make the code and dataset public. The human baseline is a nice addition because it gives context for how far the models still have to go. This setup is new in focusing specifically on crypto binaries rather than general code RE. It provides a repeatable testbed that security researchers could use to measure progress on automating parts of malware analysis or vulnerability discovery. The soft spot is the risk that models are not really doing reverse engineering. Because the challenges start from well-known algorithms, LLMs might succeed by recognizing common implementations from their training data instead of analyzing the binary structure. The abstract does not describe the exact form of the input to the models, such as raw bytes, disassembly, or decompiled code, nor does it mention any obfuscation or compiler-specific artifacts that would make pattern matching harder. Without those details, it is difficult to know if the 59 percent recovery rate would hold for more realistic binaries. The human baseline remains credible, but the gap might not translate directly to practical settings. This paper is for people who evaluate LLMs on security tasks or who want baselines for automated RE tools. A reader interested in tracking LLM capabilities in code analysis would find the task breakdown and scores useful. It is worth sending for peer review because the benchmark itself is a concrete contribution, even if the current results need more scrutiny on how the challenges were built. I recommend putting it through review with attention to the challenge construction and potential memorization issues.","headline":"CREBench gives a first structured benchmark for LLM crypto binary RE with GPT-5.4 at 59% flag recovery against humans at 92, but the challenges built from 48 textbook algorithms may let models rely on memorization instead of actual analysis.","tokens_in":2410,"tokens_out":502,"would_cite":false,"duration_ms":33343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"CREBench LLM crypto-RE benchmark has zero structural overlap with RS forcing chain or J-cost machinery","alignment":"orthogonal","rationale":"The paper's central machinery is a CTF-style benchmark built from 48 textbook cryptographic algorithms, three insecure key-usage patterns, and compiler/obfuscation difficulty tiers, evaluated via four staged tasks (algorithm ID, key extraction, wrapper reimplementation, flag recovery). This construction invokes no recognition cost J(x), golden-ratio identities, 8-tick periodicity, ratio-symmetric forcing, or parameter-free derivation of constants. It therefore lies entirely outside the RS domain; RS theorems such as reality_from_one_distinction, J-uniqueness via Aczél, and Alexander-duality D=3 forcing are neither used nor contradicted.","tokens_in":60695,"confidence":"high","tokens_out":175,"duration_ms":9096,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Large language models achieve up to 64 points on a new benchmark for cryptographic binary reverse engineering, while human experts score 92.","keywords":["large language models","cryptographic reverse engineering","benchmark","binary analysis","capture the flag","security","LLM evaluation","crypto algorithms"],"falsifier":"Observing a new LLM that achieves scores above 90 points on the CREBench challenges or empirical evidence that real-world crypto RE problems have substantially different structures than those in the benchmark would falsify the assessment of current LLM limitations.","tokens_in":2661,"feed_emoji":"🔐","tokens_out":635,"duration_ms":59751,"temperature":0.7,"pith_summary":"The paper introduces CREBench to systematically test how well large language models can reverse engineer cryptographic binaries in a capture-the-flag style. It builds 432 challenges from 48 algorithms and three insecure key usage scenarios across three difficulty levels. The evaluation shows that even the best model, GPT-5.4, scores 64 out of 100 and recovers the flag in only 59 percent of cases. This establishes a human expert baseline at 92 points, indicating that current LLMs still fall short in automating this security-critical task. A sympathetic reader would care because reverse engineering crypto code is essential for vulnerability discovery and malware analysis, and automating it could speed up security work.","feed_headline":"LLMs score 64 on crypto reverse engineering benchmark","feed_subtitle":"CREBench shows GPT-5.4 recovers flags in 59% of 432 challenges while human experts score 92, revealing a clear performance gap.","key_machinery":"CREBench, a benchmark suite of 432 challenges that requires models to identify the cryptographic algorithm, understand the logic, and recover the input flag from binary code.","core_discovery":"CREBench is a benchmark of 432 CTF challenges derived from 48 standard cryptographic algorithms and three insecure key-usage scenarios. When evaluated on four sub-tasks from algorithm identification to flag recovery, frontier LLMs reach a maximum of 64.03 points with GPT-5.4 recovering the flag in 59% of challenges, while human experts achieve 92.19 points.","pith_inferences":["Improved models trained specifically on binary disassembly data could narrow the performance gap to human levels.","CREBench could be adapted to evaluate LLMs in non-cryptographic reverse engineering domains such as general malware analysis.","If LLMs reach human-level performance, it would significantly reduce the time required for vulnerability discovery in cryptographic software.","The results imply that hybrid human-AI workflows may be the immediate practical application for crypto RE tasks."],"forward_implications":["LLMs can partially automate cryptographic reverse engineering but lag behind humans in full flag recovery.","The four sub-tasks reveal specific strengths and weaknesses in LLM performance on algorithm identification and logic analysis.","Including insecure key-usage scenarios tests the models' ability to detect common vulnerabilities in crypto implementations.","Performance gaps suggest that current LLMs are not yet ready to fully replace expert analysis in security tasks.","The benchmark provides a standardized way to track improvements in LLM capabilities for binary RE."],"fun_headline_variants":["LLMs reach 64 on CREBench cryptographic reverse engineering test","GPT-5.4 achieves 59% flag recovery in CREBench challenges","Experts score 92 against LLMs 64 in crypto RE benchmark","CREBench benchmark puts LLMs at 64 on crypto binary analysis"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 432 challenges accurately represent the distribution and difficulty of real-world cryptographic binary reverse engineering without introducing unintended biases.","fun_headline_variants_meta":{"raw":{"variants":["LLMs reach 64 on CREBench cryptographic reverse engineering test","GPT-5.4 achieves 59% flag recovery in CREBench challenges","Experts score 92 against LLMs 64 in crypto RE benchmark","CREBench benchmark puts LLMs at 64 on crypto binary analysis"]},"model":"grok-4.3","cost_usd":0.005782,"raw_usage":{"total_tokens":2770,"prompt_tokens":699,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":57824500,"prompt_tokens_details":{"text_tokens":699,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1997,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":699,"tokens_out":74,"duration_ms":17253,"temperature":1.0,"reasoning_tokens":1997,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-13T17:25:01.090989+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing a new LLM that achieves scores above 90 points on the CREBench challenges or empirical evidence that real-world crypto RE problems have substantially different structures than those in the benchmark would falsify the assessment of current LLM limitations.","supporting_citations":[],"review_version":1}