{"id":"f9e8ba98-cdd1-476b-b63e-701b9969436c","arxiv_id":"2509.05553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Contrastive fine-tuning on paired original/obfuscated code improves reverse deobfuscation for variable renaming (39-52% success) where standard fine-tuning gives near 0%, but the broad claim of bidirectional reasoning is supported for only one transformation type.","lead":"Large language models fine-tuned to obfuscate code fail to deobfuscate it, and the paper proposes contrastive fine-tuning (CFT) to recover some reverse ability without explicitly training deobfuscation. Only variable renaming shows gains (39-52% reverse success), while dead code insertion and string encryption still fail.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 39–52% reverse 'success' may measure semantics-preserving renaming, not deobfuscation: §5.0.3 never defines 'semantic success', and §4.3.2's dissimilarity/readability criteria do not require recovering the original identifiers the abstract says the model infers.","rationale":"The reader's CONDITIONAL verdict is appropriate, but I identify a different load-bearing condition. The reader's weakest assumption concerns whether positive pairs in CFT constitute implicit reverse training; that is a real philosophical and empirical question. However, the more decisive gap is the reverse-success metric. The paper's own §5.0.3 narrows the empirical claim to variable renaming, and in that narrow setting the distinction between 'recovered the original identifiers' and 'produced any readable consistent renaming' is exactly what the bidirectional-reasoning claim needs. The success criterion in §4.3.2 explicitly avoids requiring similarity to the original code, and §5.0.3 says 'semantic success' without defining it. Because variable renaming is semantics-preserving by construction, a model that simply renames variables well can pass test suites and readability checks without performing the inverse mapping the abstract describes. This concern is concrete and falsifiable: re-scoring the stored outputs with an identifier-recovery metric settles it. I therefore keep the verdict CONDITIONAL: the paper should not be rejected, but the headline claim should not be accepted until this re-scoring is reported. The positive-pair concern should also be examined, but it is secondary because the current evaluation cannot distinguish real inverse inference from a much weaker effect.","tokens_in":11842,"tokens_out":9927,"duration_ms":92353,"concrete_test":"Re-score all CFT variable-renaming outputs from §5.0.3 using alias-aware identifier recovery against the original programs: for each transformed variable, check whether the output name is an exact match or a canonical semantic-role equivalent (e.g., userIndex vs user_index) of the original name, requiring recovery of at least 80% of variables per program for success. Report exact-match and role-match rates alongside the current pass-rate numbers. If role-match recovery is near 0 while pass rate remains ~50%, the reverse result is an artifact of the §4.3.2 success criterion; if role-match recovery is high, the reverse-inference claim survives. Apply the identical scoring to the SFT outputs to confirm the 0% baseline is not a naming-format artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the 39–52% reverse 'semantic success' numbers in §5.0.3, but the paper never defines what counts as success for CFT reverse outputs. §4.3.2 defines only two proxy criteria: the output must be dissimilar from the obfuscated input (S(C_deobf, C_obf) -> 0) and readability must approach the original. For variable renaming, a transformation whose entire purpose is to preserve semantics, these criteria are satisfied by any consistent, readable renaming that compiles and passes the test suite, regardless of whether the model recovered the original identifiers or their semantic roles. Thus 52% test-passing 'deobfuscations' may be renamed programs, not inverses of the particular learned mapping; a model could score success by inventing new names (e.g., currentCount for an original userIndex) without performing the inference the abstract claims. Manual verification of a handful of plausible names is anecdotal, not a recovery rate. If the metric is this lax, the headline comparison (52% vs 0%) does not establish bidirectional reasoning; it establishes only that CFT can produce semantically valid renamed code. The reader's positive-pair objection is real but secondary: if reverse performance is not actually measured, the training-direction debate is moot.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Bidirectional Reasoning Hypothesis (BRH), which defines genuine understanding of a code transformation as the ability to invert it without explicit reverse training. The authors evaluate seven LLMs on three obfuscation transformations (variable renaming, dead code insertion, string encryption), report that standard fine-tuning (SFT) yields near-zero reverse deobfuscation, and propose Contrastive Fine-Tuning (CFT), which combines positive equivalence classification, negative classification, and forward generation (Eq. 5). They report 39–52% reverse \"semantic success\" for variable renaming under CFT versus 0% for SFT, and claim this demonstrates emergent bidirectional reasoning. Dead code insertion and string encryption do not succeed in the CFT experiments.","tokens_in":12129,"tokens_out":5315,"duration_ms":45095,"significance":"If validated, the reversibility test would be a useful operational criterion for code understanding, and CFT would be a practical training method with broad relevance. The study has notable strengths: it covers multiple models and transformation types, uses real Java programs with test-suite verification, and reports consistent SFT reverse failure across models. However, the headline result rests on an undefined success metric and lacks a base-model baseline, so the significance is conditional on a careful re-analysis of the reverse evaluation and a more precise statement of what CFT actually learns.","major_comments":[{"comment":"The reverse \"semantic success\" metric is never defined for CFT. §4.3.2 defines success via dissimilarity S(C_deobf, C_obf)→0 and readability R(C_deobf)→R(C_orig), but for variable renaming—a semantics-preserving transform—these criteria are satisfied by any readable, compiling, test-passing rename, regardless of whether the original identifiers or their semantic roles are recovered. §5.0.3 reports 39–52% \"semantic success\" without specifying the threshold or recovery criterion used. The finding therefore does not establish deobfuscation; it establishes only that CFT can produce semantically valid renamed code. Please report exact-match identifier recovery or a semantic-role recovery rate on a labeled subset as the primary reverse metric.","section":"§4.3.2, §5.0.3"},{"comment":"The claim that fine-tuning \"actively degrades pre-existing bidirectional reasoning present in base models\" is unsupported because no non-fine-tuned base model is evaluated on reverse deobfuscation. Experiment 3 tests only fine-tuned models; the observed P(T^{-1})≈0 could be the pre-existing ability of the base models rather than a degradation caused by SFT. Add a base-model reverse baseline for each transformation and prompting strategy to support the cognitive-specialization characterization.","section":"§1, §4.3.3"},{"comment":"The claim that CFT enables reverse capability \"without explicit reverse training\" is undermined by the training data construction. Positive classification pairs are (original, obfuscated) code pairs for the equivalence task L_pos, so the model is exposed to both directions of the mapping during training. This paired exposure is a form of supervision on the reverse direction, even if it is not a generation loss. The manuscript should either justify why classification exposure does not constitute reverse training or provide an ablation that removes positive pairs or uses only forward generation plus negative classification.","section":"§5.0.2"},{"comment":"The conclusions generalize from a single successful transformation. CFT achieves 39–52% reverse performance only for variable renaming; §5.0.3 states that dead code insertion and literals encryption fail. The abstract and conclusion nevertheless claim that CFT \"successfully achieved bidirectional reasoning\" and enables \"strong reverse performance\" without restricting the claim to variable renaming. Restrict the headline claims to the successful setting, or provide a substantive account of why the other transformations fail and what that implies for the general framework.","section":"Abstract, §5.0.3, §7"}],"minor_comments":[{"comment":"The abstract and §1 say six models, but §3.1 lists seven models (five open-source plus two API models); clarify the discrepancy, and state which models are included in which experiments.","section":"§1, §3.1"},{"comment":"The experimental setup refers to \"GPT-3.5-Mini\", but elsewhere the model is called GPT-3.5-Turbo; please confirm the intended model name and use it consistently.","section":"§4.2.2"},{"comment":"The text says the CFT evaluation uses \"identical Experiment 4 protocols\", but the previous experiment is labeled Experiment 3; renumber the experiments consistently.","section":"§5.0.3"},{"comment":"Reference [13] is cited for OpenAI's 50-to-1000 sample quality improvement, but [13] is the LoRA paper (Hu et al.); the reference list and in-text citations need to be corrected to match the claims about fine-tuning data sizes.","section":"§3.2.1, References"},{"comment":"The baseline CodeBLEU values between original and tool-obfuscated code are mentioned in the text (e.g., S(C_obf, C_orig)≈0.22, 0.33) but are not given consistently in a table or figure legend; please add these baselines so the dissimilarity thresholds are interpretable.","section":"§4.1.3, Figure 3"},{"comment":"The manuscript provides no statement about code or data availability; please add a reproducibility section detailing prompts, hyperparameters, and evaluation scripts.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the reverse success metric: as written, the 39–52% figure may measure valid renaming rather than deobfuscation, and the 'without explicit reverse training' claim is weakened by the positive-pair construction. These are fixable with re-analysis and more careful framing, so the paper is arguably revisable. I would ask the authors to re-evaluate reverse outputs against original identifiers or semantic roles, add base-model reverse baselines, and scope the conclusions to variable renaming. The paper is not yet defensible in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — you should know this paper has a promising idea and a load-bearing measurement flaw. The idea: fine-tune on forward obfuscation plus contrastive equivalence pairs, and see whether reverse deobfuscation emerges. The headline numbers look striking (SFT 0% vs CFT 39–52% on variable renaming), but the success metric is never defined, and as defined elsewhere in the paper it does not actually require recovering the original identifiers. For variable renaming, any readable rename that compiles and passes tests counts as 'semantic success,' so the 52% might just be 'model produces valid renamed code,' not 'model infers the inverse mapping.' That is the crux.\n\nWhat is legitimately good: the forward obfuscation study is reasonably thorough — seven models, three transformations, error decomposition, self-correction loops, CodeBLEU and readability analysis. The 'cognitive specialization' observation (forward fine-tuning helps forward, hurts reverse) is plausible, and the contrastive training recipe is a reasonable response. The authors also honestly report that only one of three transformations works; the conclusion admits the scope.\n\nThe soft spots beyond the metric: (1) the positive-classification training pairs are literally (original, obfuscated) pairs, so the model sees both directions of the mapping — calling the reverse result 'without explicit reverse training' is misleading. (2) No base-model reverse baseline, so the claim that fine-tuning degrades pre-existing ability is not supported. (3) The BFT baseline is mentioned in the setup but absent from results. (4) 'First to apply contrastive learning to code comprehension' is wrong; that literature is large and uncited. (5) No artifacts.\n\nNone of these are fatal to the underlying question — the CFT effect on variable renaming might be real and worth understanding. But the paper as written overclaims to all transformations, and the central metric needs to be tightened to exact-match recovery or identifier-level accuracy before the headline means anything.\n\nWho is this for? Researchers working on code LLM training and evaluation of 'understanding' would get something from the setup and the forward experiments. I would send it to peer review, but with a clear directive to fix the evaluation and add baselines. My own verdict: not citable in current form.","headline":"A promising idea with a measurement flaw: the 52% reverse 'success' may measure valid renaming, not recovered inverse mapping.","tokens_in":12678,"tokens_out":1902,"would_cite":false,"duration_ms":16404,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Contrastive Fine-Tuning, trained only on forward obfuscation and equivalence classification, lets code models reverse the transformation without reverse examples—deobfuscation rises from 0% to 39–52% on variable…","keywords":["bidirectional reasoning","contrastive fine-tuning","code obfuscation","deobfuscation","semantic understanding","large language models","cognitive specialization","code transformation"],"falsifier":"Train a CFT variant with the forward generation and negative-pair losses unchanged but replace the positive-equivalence task with pairs of unrelated but semantically equivalent programs, so no original–obfuscated mapping is shown; if reverse deobfuscation on the held-out evaluation set stays near 39–52%, pairing was not the mechanism, and if it falls toward 0%, the paper's 'without explicit reverse training' claim is refuted.","tokens_in":11691,"feed_emoji":"🔄","tokens_out":10393,"duration_ms":82522,"temperature":0.7,"pith_summary":"This paper argues that genuine understanding of a code transformation should be reversible: a model that learns to obfuscate should be able to deobfuscate without ever seeing reverse examples. It documents a failure mode it calls cognitive specialization, in which standard fine-tuning on forward obfuscation improves forward performance but leaves reverse deobfuscation at roughly 0%, across all tested models and prompting strategies. The proposed remedy, Contrastive Fine-Tuning (CFT), trains on three objectives at once: recognizing when original and obfuscated code are semantically equivalent, recognizing when code has different meaning, and generating obfuscated code in the forward direction. On variable renaming, CFT raises reverse success from 0% to 39–52% on GPT-3.5-Turbo, GPT-4.1-Mini, and QwenCoder while preserving forward quality; dead code insertion and string encryption do not show the same emergence. If the claim is right, bidirectional reasoning is both a practical training design and a behavioral test that separates semantic understanding from surface pattern matching.","feed_headline":"Contrastive training lets LLMs undo obfuscation they never learned","feed_subtitle":"Reverse deobfuscation jumps from 0% to 39–52% on variable renaming when trained with contrastive pairs.","key_machinery":"The load-bearing mechanism is Contrastive Fine-Tuning (CFT), a combined loss $\\mathcal{L}_{\\mathrm{CFT}} = \\mathcal{L}_{\\mathrm{pos}} + \\mathcal{L}_{\\mathrm{neg}} + \\mathcal{L}_{\\mathrm{gen}}$ over balanced triplets: positive pairs (an original program and its obfuscated version, labeled semantically equivalent), negative pairs (programs with different functionality), and forward obfuscation generation examples. The classification tasks force the model to represent what a transformation preserves and what it changes, while the generation task preserves forward skill; the paper's central claim is that this joint pressure lets the inverse operation $T^{-1}$ emerge without any reverse examples. The Bidirectional Reasoning Hypothesis supplies the evaluation criterion: for a semantics-preserving transformation $T$, genuine understanding requires success on both $T$ and $T^{-1}$, and the paper operationalizes that symmetry as the test that separates understanding from pattern matching.","core_discovery":"The paper's central discovery is that the directional bias created by fine-tuning is not fixed. Standard fine-tuning on obfuscation produces a sharply asymmetric model: forward transformation success exceeds 85% while reverse deobfuscation success is effectively zero, and advanced prompting—few-shot, chain-of-thought, or both—does not recover the inverse operation. Training the same models with Contrastive Fine-Tuning (CFT) changes this: with only forward obfuscation examples plus positive and negative semantic-equivalence classification, reverse deobfuscation emerges at 39–52% for variable renaming in the GPT models and QwenCoder, with manual checks showing meaningful identifiers rather than copied obfuscated tokens. The paper reads this as evidence that contrastive supervision builds a semantic representation from which the inverse transformation can be derived, and that the standard fine-tuning objective instead collapses the model onto a unidirectional mapping.","pith_inferences":["Editorial inference: the same triplet recipe should be tested on other invertible code transformations, such as formatting changes, comment removal, or refactoring renames, which have cleaner ground truth and would show whether the effect generalizes beyond obfuscation.","Editorial inference: the decisive control for the 'without explicit reverse training' claim is to train on the forward generation loss without the positive-equivalence head; if reverse success then collapses toward 0%, the original–obfuscated pairs in the positive task were themselves the reverse training signal.","Editorial inference: the paper's negative results predict that harder transformations will require stronger contrastive signals, such as hard negatives or an explicit cycle-consistency loss, before reverse ability appears; this is directly testable with the same 300-sample evaluation set."],"forward_implications":["Forward-only fine-tuning on code transformations should be re-evaluated for reverse performance, since high forward scores can hide the loss of reversible understanding.","CFT's jump from 0% to 39–52% reverse success on variable renaming provides a concrete baseline for any future training method that claims bidirectional reasoning.","The absence of the effect on dead code insertion and string encryption implies that transformation complexity, not just exposure to pairs, gates whether reverse ability emerges.","The capacity-dependent pattern—GPT models and QwenCoder succeed while other 7B–15B open models do not—implies that the contrastive objective alone is not sufficient and that model scale or pretraining matters."],"supporting_citations":[{"why":"Supplies the contrastive learning objective that CFT adapts from vision to code.","marker":"[4]"},{"why":"Provides the 10,000 Java programs used as the base for training pairs.","marker":"[21]"},{"why":"Obfuscation tool that generates the three transformation types and the tool-reference obfuscation.","marker":"[25]"},{"why":"CodeBLEU is the primary syntactic similarity metric for forward and reverse evaluations.","marker":"[22]"},{"why":"LoRA is the parameter-efficient fine-tuning method used on the open-source models.","marker":"[13]"},{"why":"Contributes the 300 curated evaluation samples with verified test coverage.","marker":"[16]"},{"why":"Cycle-consistency baseline that relies on explicit forward and backward training, contrasting with CFT's claim of no reverse examples.","marker":"[12]"},{"why":"Catastrophic forgetting result that motivates the cognitive-specialization interpretation.","marker":"[15]"}],"fun_headline_variants":["Contrastive fine-tuning gives LLMs reverse obfuscation skills","LLMs learn to deobfuscate from contrastive training alone","Bidirectional reasoning emerges from contrastive pairs in LLMs","Reverse obfuscation jumps from 0% to 52% with contrastive training","Contrastive learning unlocks bi-directional reasoning in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the positive-equivalence task, which Section 5.0.2 constructs from paired originals and their obfuscated versions, is not a form of reverse training; if that pairing counts as reverse exposure, the claimed 'emergence without explicit reverse training' collapses into a representation-alignment effect.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive fine-tuning gives LLMs reverse obfuscation skills","LLMs learn to deobfuscate from contrastive training alone","Bidirectional reasoning emerges from contrastive pairs in LLMs","Reverse obfuscation jumps from 0% to 52% with contrastive training","Contrastive learning unlocks bi-directional reasoning in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1362,"prompt_tokens":979,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":595,"tokens_out":383,"duration_ms":3351,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:14.377929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CFT variant with the forward generation and negative-pair losses unchanged but replace the positive-equivalence task with pairs of unrelated but semantically equivalent programs, so no original–obfuscated mapping is shown; if reverse deobfuscation on the held-out evaluation set stays near 39–52%, pairing was not the mechanism, and if it falls toward 0%, the paper's 'without explicit reverse training' claim is refuted.","supporting_citations":[{"cited_title":"Java obfuscator (gui)","cited_arxiv_id":null,"evidence_quote":"Obfuscation tool that generates the three transformation types and the tool-reference obfuscation."},{"cited_title":"Condefects: A large-scale dataset of real-world concurrency bugs and fixes","cited_arxiv_id":null,"evidence_quote":"Contributes the 300 curated evaluation samples with verified test coverage."},{"cited_title":"Unpaired image-to- image translation using cycle-consistent adversarial networks.Proceedings of the IEEE International Conference on Computer Vision, pages 2223–2232, 2017","cited_arxiv_id":null,"evidence_quote":"Cycle-consistency baseline that relies on explicit forward and backward training, contrasting with CFT's claim of no reverse examples."},{"cited_title":"Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al","cited_arxiv_id":null,"evidence_quote":"Catastrophic forgetting result that motivates the cognitive-specialization interpretation."}],"review_version":2}