{"id":"59145de1-6219-4916-b54a-61acf496d372","arxiv_id":"2608.06960","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A nine-dimensional metric with hand-tuned weights compares decompiled code to original OpenWrt programs; recompilable outputs score higher, but one component already encodes recompilation failure.","lead":"This paper proposes a nine-dimensional score for measuring how faithfully decompiled code matches the original binary, applied to 19,625 outputs from five decompilers. It finds that behavioral and structural similarity separate recompilable from non-recompilable code, but part of the metric already builds recompilation failure into the score.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline d=0.96 for behavioral similarity is partly definitional because S_api is forced to 0 on recompilation failure, and the paper's own numbers contradict it.","rationale":"The reader identified the load-bearing weakness: S_api is defined to be zero on recompilation failure, which makes the behavioral similarity difference between success and failure groups partly definitional. Independent review confirms this from the paper's equations and numbers. Section 3.3.2 explicitly imposes the zero penalty, Eq. (8) folds S_api into S_behav with weight 0.30, and the paper's own Table 6 reports failure-group behavioral means that exceed the maximum possible under that rule. This internal contradiction invalidates the primary empirical evidence for d=0.96 and the overall d=0.92. The sub-metric pattern is also consistent with the concern: API detection, the component forced to zero on failure, carries the largest effect (d=1.91), while the two other behavioral sub-metrics show inverse effects. A concrete recomputation test can settle whether the stated rule was applied; either outcome undermines the central claim as presented. Therefore the correct verdict remains REJECT, and no adjustment to the reader's assessment is needed.","tokens_in":14016,"tokens_out":1594,"duration_ms":19098,"concrete_test":"Re-run the analysis with S_api computed normally for recompilation-failed samples instead of being forced to zero, then recompute Table 4, Table 6, and d statistics. If the behavioral d drops substantially or the high-behavior failure group no longer has mean 0.814, the reported d=0.96 is confirmed to be an artifact of the zero rule. Additionally, inspect the scored data for a sample of failed recompilations: if any failed sample has S_behav > 0.70, the zero rule was not applied as documented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that behavioral similarity predicts recompilation success rests on S_behav, defined in Eq. (8) as 0.40*S_logic + 0.30*S_api + 0.30*S_error. Section 3.3.2 states: 'If recompilation fails, the score is 0' for API/system-call correctness. Since recompilation success/failure is the outcome variable, failure mechanically lowers S_api and therefore S_behav. The reported d=0.96 thus conflates the metric with the outcome it is supposed to predict, making the headline effect partly definitional rather than evidential. The contradiction is visible in the paper's own data: Table 6 reports a high-behavior failure group with mean behavioral similarity 0.814, but if S_api=0 for all failures and S_logic and S_error are each at most 1.0, the maximum possible S_behav is 0.70. Either the zero rule was not actually applied (in which case the text and the reported scores are inconsistent), or it was applied (in which case the failure-group mean of 0.728 in Table 4 is impossible and the d values are artifacts of the scoring rule). The sub-metric analysis in Section 4.2.4 compounds the problem: API detection shows d=1.91, exactly the component that is zeroed on failure, while logic operations and error handling show inverse effect sizes. The paper acknowledges some of this, but the acknowledgment does not address the definitional entanglement of the primary result. Without resolving this contradiction, the claim that behavioral similarity is an important predictor of decompilation quality is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a nine-dimensional quality metric for decompiled C code, organized into structural, behavioral, and semantic similarity, and uses it to compare recompilation-success versus recompilation-failure groups across 318 OpenWrt programs and five decompilation methods. The authors report large effect sizes for overall score (d=0.92) and behavioral similarity (d=0.96), analyze sub-metrics, compiler options, prompt strategies, model differences, and existing metrics, and identify a 'high-behavior failure' group to argue that structural similarity captures defects missed by behavioral evaluation.","tokens_in":14380,"tokens_out":5294,"duration_ms":51632,"significance":"A validated multi-dimensional decompilation quality metric would be valuable for IoT vulnerability analysis and for evaluating LLM-generated code, so the paper addresses an important problem. The study has notable strengths: a large dataset (nominally 19k+ generated programs), automated metric computation, comparison with CodeBLEU and CodeScore, and the high-behavior-failure analysis. However, the primary statistical claim is currently not interpretable because one component of the behavioral metric is assigned a score of 0 when recompilation fails, which makes the outcome partly definitional; moreover, the paper's own reported numbers contradict this rule. The metric framework and dataset may be useful after correction, but the present evidence does not establish the headline predictive claim.","major_comments":[{"comment":"The rule in Section 3.3.2, 'If recompilation fails, the score is 0,' applied to S_api, combined with Eq. (8), implies that for every failed recompilation S_behav <= 0.4*1.0 + 0.3*0 + 0.3*1.0 = 0.7. Yet Table 4 reports a failure-group behavioral similarity of 0.728, and Table 6 reports the high-behavior failure group's behavioral similarity as 0.814. These values are impossible unless the zero rule was not actually applied. The text and the numbers cannot both be true, so the reported d=0.96 cannot be interpreted as evidence that behavioral similarity predicts recompilation success.","section":"Section 3.3.2, Eq. (8), Tables 4 and 6"},{"comment":"The sub-metric analysis shows API detection with d=1.91, while logic operations and error handling have inverse effect sizes (d=-0.49 and d=-0.35). Because S_api is zeroed on recompilation failure, the large API effect is definitional rather than evidential, and the remaining behavioral sub-metrics actually oppose the headline direction. The claim that 'behavioral similarity is an important predictor' is therefore largely an artifact of including S_api in S_behav. A corrected analysis should remove the zero rule or report S_behav excluding S_api.","section":"Section 4.2.4, Table 5"},{"comment":"The sample counts are mutually inconsistent. The abstract says 19,625 results; Section 4.2.1 says 21,624 generated programs, with 2,299 generation errors and 402 unmeasurable cases, leaving 19,223; Section 4.2.2 reports 2,736 successes out of 19,625; Table 3's per-model counts sum to 19,625; Figure 3 and Table 6 use 19,223 (2,702 success and 16,521 failure); and Section 4.2.7's F-statistic denominator implies 19,625. These inconsistencies affect every reported effect size and must be reconciled, and the statistical analyses must be re-run on a single, clearly stated sample.","section":"Sections 4.2.1, 4.2.2, Table 3, Figure 3, Table 6"},{"comment":"The 'high-behavior failure' analysis is used to argue that structural similarity is complementary to behavioral similarity, but the group's mean behavioral similarity of 0.814 is impossible under the scoring rule stated in Section 3.3.2. The complementary-relationship claim cannot be evaluated until the behavioral scores are recomputed under a consistent rule. In addition, the thresholds of 0.8 (behavioral) and 0.6 (structural) appear to be introduced post hoc and should be justified independently of the observed data.","section":"Section 4.2.5, Table 6"}],"minor_comments":[{"comment":"The claim that the score distribution is 'approximately normal' is not supported by a normality test or diagnostic plots; given the bounded 0-1 scale and the reported mean/median near 0.6, this should be substantiated or rephrased.","section":"Section 4.2.1, Figure 2"},{"comment":"The phrase 'If recompilation fails, the score is 0' is ambiguous about whether it zeroes the entire API/system-call score or only one component. Please specify exactly which term in the computation is affected and how this interacts with the SequenceMatcher and Jaccard components.","section":"Section 3.3.2"},{"comment":"Reporting standard deviations or confidence intervals alongside the group means would help readers assess the overlap between the success and failure distributions, especially for the sub-metric comparisons with small or inverse effect sizes.","section":"Tables 4 and 6"}],"recommendation":"major_revision","confidential_remarks":"The internal contradiction between Section 3.3.2 and Tables 4/6 is severe enough that a simple textual revision is insufficient; the metric computation and all dependent statistics must be redone. If the authors cannot provide corrected numbers and a revised interpretation, rejection would be appropriate. The current manuscript should not be accepted in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds something worth talking about: a nine-dimensional quality metric for decompilation output, applied to 19k samples from five decompilers. The three-category split (structural, behavioral, semantic) is reasonable, and the idea of looking for high-behavior failures that still don't recompile is genuinely interesting. The comparison against CodeBLEU and CodeScore is also useful context.\n\nBut the central claim does not survive contact with the metric definition. Section 3.3.2 says API/system-call correctness scores 0 when recompilation fails, and Eq. (8) folds that into S_behav. Since recompilation success is the outcome variable, the behavioral score is partly measuring the thing it claims to predict. The d=0.96 for behavioral similarity is therefore at least partly an artifact.\n\nThe paper's own numbers make this worse. If the zero rule is applied, the maximum possible S_behav for any failure is 0.70. Yet Table 4 reports failure-group behavioral similarity of 0.728, and Table 6 reports a high-behavior failure group with 0.814. Both are impossible. So either the rule was not actually applied, in which case the text is wrong, or it was applied, in which case the tables are wrong and the effect sizes are definitional. Either way, the main empirical result is unsupported.\n\nThere are also sample-count inconsistencies: the abstract says 19,625, Section 4.2.1 says 19,223 analyzed, and Figure 3's group sizes (16,521 / 2,702) don't match Table 4's (16,889 / 2,736). The two sub-metrics that show inverse effects (logic operations d=-0.49, error handling d=-0.35) are mentioned in the limitations, but that doesn't fix the definitional entanglement. No artifacts or code are released, and the weight tuning is explicitly ad hoc.\n\nThe framework itself is plausible, and the high-behavior failure analysis could be valuable once the scoring contradiction is resolved. But as it stands the causal claims are not supported. I would not cite it, and I would not bring it to reading group. It does deserve a serious referee, though, because the topic is important and the flaws are identifiable and fixable. My recommendation: send it to review, but the referee should push hard for a corrected scoring rule and a rewriting of the statistical claims.","headline":"Useful metric framework, but the headline d=0.96 is partly definitional because the behavioral score is zeroed on recompilation failure, and the paper's own tables contradict that rule.","tokens_in":14934,"tokens_out":2294,"would_cite":false,"duration_ms":23219,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new nine-dimensional score measures how faithfully decompiled IoT code preserves structure, behavior, and semantics rather than just whether test cases pass.","keywords":["Decompilation","Large Language Models","Program Equivalence","IoT Security","Statistical Evaluation","Binary Analysis","Structural Similarity","Behavioral Similarity"],"falsifier":"Recompute $S_{\\mathrm{behav}}$ for the 700 high-behavior failure cases under the paper's stated rule that the API/system-call sub-metric is 0 on recompilation failure; the maximum possible behavioral score would then be 0.7, so the reported mean of 0.814 directly shows whether the rule was applied or whether the effect sizes are partly definitional artifacts.","tokens_in":13799,"feed_emoji":"🔍","tokens_out":15571,"duration_ms":128311,"temperature":0.7,"pith_summary":"Decompilation of IoT firmware is meant to reconstruct code that behaves like the original binary, but current evaluation metrics concentrate on whether test cases pass and can miss code that looks plausible while its internal control flow or error handling has changed. The paper proposes a nine-dimensional quality score spanning structural similarity (function signatures, control-flow graph, data structures), behavioral similarity (logic operations, API/system calls, error handling), and semantic similarity (identifiers, variable names, type use). On 19,223 decompiled outputs from 318 OpenWrt programs, it finds that outputs that recompile score higher overall (Cohen's $d=0.92$), with behavioral similarity the strongest single predictor ($d=0.96$) and structural similarity also substantial ($d=0.69$). It isolates 700 cases that failed to recompile despite behavioral similarity of 0.8 or higher, and those cases have lower structural similarity than the recompilation-success group, which the paper takes as evidence that structural fidelity is an independent axis that behavioral checks cannot replace. The result matters because vulnerability analysts need to know whether decompiled code they are reading can be trusted in the details where IoT vulnerabilities actually hide.","feed_headline":"Nine metric axes predict which decompiled firmware actually recompiles","feed_subtitle":"Structural and behavioral scores, not test passing, catch the failures in 19,223 decompiled OpenWrt programs.","key_machinery":"The load-bearing object is the weighted nine-dimensional score. The three category aggregates are $S_{\\mathrm{struct}} = 0.30S_{\\mathrm{sig}} + 0.50S_{\\mathrm{cfg}} + 0.20S_{\\mathrm{data}}$, $S_{\\mathrm{behav}} = 0.40S_{\\mathrm{logic}} + 0.30S_{\\mathrm{api}} + 0.30S_{\\mathrm{error}}$, and $S_{\\mathrm{sem}} = 0.50S_{\\mathrm{ident}} + 0.30S_{\\mathrm{var}} + 0.20S_{\\mathrm{type}}$, combined into $S_{\\mathrm{total}} = 0.35S_{\\mathrm{struct}} + 0.40S_{\\mathrm{behav}} + 0.25S_{\\mathrm{sem}}$. The structural axis compares function signatures, data-structure restoration, and, with the highest weight, a control-flow-graph (CFG) similarity computed from normalized differences in node count, edge count, and cyclomatic complexity. The behavioral axis accumulates severity-weighted counts of logic, API/system-call, and error-handling defects, with API correctness formed from order-preserving sequence matching plus a Jaccard set-overlap coefficient. The semantic axis covers magic-number avoidance, variable-name agreement, and type-system usage. Because the dataset is so large that ordinary $p$-values are trivially significant, the analysis uses Cohen's $d$ effect size as the criterion for whether a metric distinguishes the recompilation-success group from the failure group.","core_discovery":"The paper's central claim is that decompilation quality for vulnerability analysis is not a single axis: recompilability and behavioral equivalence are necessary but not sufficient, and the internal structure of restored code has to be scored explicitly. It defines a nine-dimensional metric and applies it to 19,223 decompilation outputs from 318 OpenWrt programs produced by one rule-based method and four LLM-based methods. Recompilation-successful outputs beat failed ones on overall score with Cohen's $d=0.92$; behavioral similarity separates the groups most strongly ($d=0.96$), structural similarity follows ($d=0.69$), and semantic similarity is comparatively weak ($d=0.33$). The decisive evidence is the high-behavior failure group: 700 outputs whose behavioral similarity (0.814) exceeded the success group's (0.787) yet failed to recompile, while their structural similarity (0.549) fell below the success group's (0.603). The paper reads this as showing that API-level and test-level agreement can look correct while defects in branch structure or error-handling paths are what actually break recompilation, and that only a structural axis can detect those.","pith_inferences":["A testable extension is to turn the two axes into a classifier: measuring precision and recall of a behavioral-threshold/structural-threshold rule (for example, behavior at least 0.8 and structure at least 0.6) for predicting recompilation would quantify how much the reported group differences help in practice; the paper reports group means but not classification performance.","Because the API/system-call sub-metric alone has $d=1.91$ while the other sub-metrics are small or inverse, most of the behavioral signal may be carried by API-call fidelity; a follow-up could test whether a one-dimensional API-call metric predicts recompilation almost as well as the full nine-dimensional score.","The weights assigned to the three categories and nine sub-metrics are chosen by the authors and not derived from data; re-weighting with equal weights or with weights fitted to predict recompilation could change the reported effect sizes, and the paper itself flags this as open.","If the metric is intended for vulnerability detection, the next check would be correlating low structural scores in decompiled firmware with known CVE-class defects such as missing NULL checks and leak-on-error paths, which the paper motivates but does not measure."],"forward_implications":["Vulnerability analysts can use the nine-dimensional score, or its structural sub-scale, to rank decompiled firmware by how faithful the internal structure is before manual review.","Behavioral similarity alone is not enough: the 700-case high-behavior failure group shows that a decompiled function can match API calls and logic yet still fail to compile because of branch and error-handling structure.","General-purpose LLMs (ChatGPT-5.1, Claude Haiku 4.5, DeepSeek-R1) score about 0.1 higher than Ghidra and LLM4Decompile, suggesting that current LLM-based decompilation is the more promising route for faithful restoration.","Prompt strategy and program category have negligible effect on quality ($\\eta^2=0.006$ and $\\eta^2=0.023$), so improvements should target model choice and metric-driven training rather than prompt engineering.","Higher compiler optimization levels lower recompilation success (15.3% at -O0 to 12.6% at -O3), so optimization-induced structure loss is a concrete obstacle for practical firmware analysis."],"supporting_citations":[{"why":"Defines the decompilation goal of producing code functionally identical to the original binary, the normative baseline that the proposed metric operationalizes.","marker":"(Cifuentes and Gough, 1995)"},{"why":"Introduces DeGPT and the MSSC behavioral-similarity metric that the paper extends with structural dimensions.","marker":"(Hu et al., 2024)"},{"why":"Introduces D-LiFT and D-Score, the correctness-plus-readability baseline that the paper contrasts with structural evaluation.","marker":"(Zou et al., 2025)"},{"why":"Provides LLM4Decompile, one of the five decompilation methods whose outputs are scored in the experiments.","marker":"(Tan et al., 2024)"},{"why":"Reports empirical semantic defects in rule-based decompiler output, motivating the need for structural similarity.","marker":"(Sirlanci et al., 2025)"},{"why":"Describes DecompileBench, the latest benchmark that the paper says still omits structural similarity.","marker":"(Gao et al., 2025)"},{"why":"Defines CodeBLEU, an existing code-generation metric used as a comparison baseline.","marker":"(Ren et al., 2020)"},{"why":"Justifies using Cohen's $d$ effect size rather than $p$-values for the large-sample group comparisons.","marker":"(Sullivan and Feinn, 2012)"},{"why":"Defines cyclomatic complexity, one of the three quantities used in the control-flow-graph structural similarity.","marker":"(McCabe, 1976)"}],"fun_headline_variants":["Structural scores catch decompilation failures that behavior misses","Nine-axis metric predicts which decompiled firmware recompiles","Behavioral similarity alone misses decompiled code that won't recompile"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that the behavioral-similarity score is computed independently of the recompilation outcome, even though the paper's own rule (Section 3.3.2) sets the API/system-call sub-metric to zero on recompilation failure and folds that zero into the behavioral score.","fun_headline_variants_meta":{"raw":{"variants":["Structural scores catch decompilation failures that behavior misses","Nine-axis metric predicts which decompiled firmware recompiles","Behavioral similarity alone misses decompiled code that won't recompile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2569,"prompt_tokens":1081,"completion_tokens":1488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":1435}},"tokens_in":697,"tokens_out":1488,"duration_ms":13481,"temperature":1.0,"reasoning_tokens":1435,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:34:55.993341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute $S_{\\mathrm{behav}}$ for the 700 high-behavior failure cases under the paper's stated rule that the API/system-call sub-metric is 0 on recompilation failure; the maximum possible behavioral score would then be 0.7, so the reported mean of 0.814 directly shows whether the rule was applied or whether the effect sizes are partly definitional artifacts.","supporting_citations":[],"review_version":1}