{"id":"fe167833-0096-4e69-9583-d267114b7446","arxiv_id":"1908.06895","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"No single Java decompiler produces recompilable, behavior-preserving source code for more than 84% (syntactic) and 78% (test-based semantic) of the studied classes, while a naive multi-decompiler ensemble reaches 95% and 91%.","lead":"This paper measures how well eight Java decompilers reconstruct source code from compiled bytecode, using 14 real projects and 2041 classes. It finds that even the best decompiler fails on a substantial share of classes, and that combining several decompilers improves success rates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic equivalence rates depend on unquantified test coverage; the 78% and 91% figures could be optimistic, though the no-single-decompiler claim likely survives.","rationale":"I read the paper as an empirical comparative study whose main contribution is the automated EMI-based semantic assessment and the resulting rates. The central qualitative claim (no single decompiler recompiles more than about 85%) is robust because it rests on syntactic correctness. The quantitative semantic claim (Procyon 78%, Multi-DC 91%) depends on Definition 3, which equates semantic equivalence with passing the original test suite. The paper limits the denominator to 2,397 test-covered class files but does not report branch or method coverage per class; a class can be counted as covered by a single test that exercises only one path. The authors explicitly acknowledge this in Section V-a, but they do not quantify or bound the bias. This is the most load-bearing weakness because the headline numbers are the paper's distinctive quantitative contribution. I considered whether the Multi-DC selection rule (stop at first recompilable output) is a more serious flaw; it means Multi-DC inherits Procyon's deceptive outputs, but it is labeled 'naive' and does not threaten the paper's main claim, so I do not elevate it. The proposed differential test with generated inputs would settle whether the coverage gap actually moves the rates; until then, a conditional verdict is appropriate. No change to the reader's verdict is needed.","tokens_in":15546,"tokens_out":8225,"duration_ms":86013,"concrete_test":"Take a random sample of 100 of the 2,397 test-covered class-compiler pairs for which Procyon's output passes the original tests. For each, use a feedback-directed test generator (e.g., Randoop or EvoSuite) to produce 50-100 new unit tests that exercise methods in the decompiled class; run those tests against both the original bytecode and the recompiled-decompiled bytecode, keeping only tests that pass on the original, and record any behavioral divergence. If the divergence rate exceeds roughly 5%, the 78% semantic-equivalence rate is materially optimistic and should be re-reported as a lower bound conditioned on original test coverage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's semantic claims rest on Definition 3 (Section III-A): a decompiled program is semantically equivalent modulo inputs if it passes the original project's test suite. The study then narrows the semantic evaluation to the 2,397 class files covered by at least one test (Section IV-B, Figure 3). This is the load-bearing step: the 78% figure for Procyon and the 91% figure for Multi-DC are pass rates on existing tests, not measurements of behavioral equivalence, and the paper does not quantify test adequacy per class. A class can be counted as covered when a test merely touches its constructor or one method; the decompiler may have changed behavior in every other method and the pipeline will still label it equivalent. The internal-validity paragraph (Section V-a) acknowledges that 'the coverage and quality of the test suite ... influences our observations' but does not bound the effect. Because the headline semantic rates are the paper's main quantitative contribution, an unquantified coverage gap is the most serious threat to the specific numbers, even though the qualitative claim that no single decompiler recompiles everything would survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a fully automated empirical evaluation of eight Java bytecode decompilers (CFR, Dava, Fernflower, JADX, JD-Core, Jode, Krakatau, Procyon) on 14 open-source projects totaling 2,041 Java classes. The pipeline compiles each project with javac and ecj, decompiles the resulting class files, and assesses output along four dimensions: syntactic correctness (recompilability), syntactic distortion (AST edit distance from the original), strict bytecode equivalence, and semantic equivalence modulo inputs (passing the original project's test suite on test-covered classes). The main findings are that no single decompiler handles all classes: Procyon achieves the best single-tool results with about 84% syntactically correct and 78% semantically equivalent outputs, while a naive ensemble (Multi-DC) raises those figures to about 95% and 91%. The authors also report deceptive decompilation cases and manually triaged several decompiler bugs.","tokens_in":15620,"tokens_out":7888,"duration_ms":74640,"significance":"If the results hold, this is a useful and substantial empirical contribution: it is an order of magnitude larger than prior decompiler comparisons, it adapts equivalence-modulo-inputs testing to decompilation, and it provides a publicly available, reproducible pipeline and dataset. The central qualitative claim that no single modern decompiler can correctly handle the variety of real-world Java bytecode is well supported by the recompilation and test-passing measurements. The paper's main quantitative contribution, however, is the semantic-equivalence rates, and those are directly bounded by the quality and coverage of the original test suites. The authors acknowledge this threat but do not quantify it, so the exact figures should be treated as upper bounds rather than precise measurements of behavioral equivalence. The syntactic-correctness results and the Multi-DC comparison are more robust and likely to survive even a stricter semantic evaluation.","major_comments":[{"comment":"The semantic-equivalence metric is defined as passing the original test suite, and the RQ2 evaluation is restricted to the 2,397 class files covered by at least one test. The paper does not quantify per-class test adequacy: a class counts as covered if a test touches it at all, so a decompiler may alter untested methods and still be labeled equivalent. Consequently, the headline rates (Procyon 78%, Multi-DC 91%; Figure 3 and Table III) are upper bounds under the current test suites, and the amount by which they would change under stronger coverage is unknown. The internal-validity paragraph acknowledges that test-suite quality influences the observations, but it does not bound the effect. Please report coverage statistics (e.g., method/line coverage per class and the distribution of the number of tests per class) and, if possible, recompute the rates on classes meeting a minimum coverage threshold.","section":"§III-A, Definition 3; §IV-B, Figure 3; §V-a"},{"comment":"The summary table mixes denominators: the #Recompilable ratios are computed over all class-compiler pairs (2,041 javac pairs plus 1,887 ecj pairs, i.e., 3,928), while the #PassTest ratios are computed only over the 2,397 test-covered pairs. The table gives no denominator row or note, so a reader cannot tell that the 84% and 78% figures refer to different populations. In addition, the Procyon counts in the text (1,609/1,887 = 85.3% for javac; 1,532/1,887 = 81.2% for ecj) do not match Figure 2 (85.7% and 81.2%, respectively). Please state the denominators explicitly in Table III and reconcile the RQ1 numbers between the text and the figure.","section":"§IV-A, Figure 2, Table III"},{"comment":"Definition 2 defines syntactic distortion as the minimum number of atomic edits to transform the AST of the original into the AST of the decompiled version, which includes renaming edits; however, Section IV-D and Figure 5 state that renaming is excluded from the measurement. Since variable renaming is one of the most visible differences between decompilers, the reported distortion values understate the dissimilarity and do not correspond to the definition as written. The definition or the experimental procedure should be made consistent, and the metric description should clearly state whether identifier changes contribute to the edit distance.","section":"§III-A, Definition 2; §IV-D, Figure 5"}],"minor_comments":[{"comment":"The conclusion says the projects were 'compiled with 2 different decompilers'; this should say 'compilers'.","section":"§VII, Conclusion"},{"comment":"The statement 'This represents 1887 class files for each compiler' is ambiguous because commons-lang was compiled only with javac; please state the per-compiler denominators explicitly (2,041 for javac, 1,887 for ecj).","section":"§III-D, Study Subjects"},{"comment":"The text reports p-values for the chi-square test but mentions a 99% confidence level; it would be clearer to state the significance level and interpret the p-values directly against it.","section":"§IV-A, RQ1"},{"comment":"The bars show counts but not percentages; adding percentage labels or a second axis would make the ranking by semantic-equivalence rate easier to read, especially since the denominator (2,397) is fixed.","section":"Figure 3"},{"comment":"The phrase 'We exclude Dava and Krakatau because they that do not handle correctly any unique class file' contains a grammatical error and should be rephrased as 'because they do not handle any unique class file correctly.'","section":"§IV-E, RQ5"},{"comment":"The #LOC column includes Python files for Krakatau, but the column heading says 'line of code in all Java files'; clarify that the count is language-inclusive or rename the column.","section":"Table I"},{"comment":"Reference [17] has a malformed URL ('https://http://java-decompiler.github.io/'); it should be 'https://java-decompiler.github.io/'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The coverage limitation is real but fixable with additional sensitivity analysis, and the artifact and reproducibility are clear strengths. I see no concerns about novelty or authorship practices; the main issue is that the headline semantic-equivalence rates need to be qualified by test-coverage sensitivity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution. The paper adapts equivalence modulo inputs (EMI) from compiler validation to decompiler evaluation and runs it at a scale roughly an order of magnitude beyond earlier hand-crafted studies: 8 decompilers, 14 projects, 2041 classes, two compilers. The measurement pipeline is public, the definitions are explicit, and the authors actually reported bugs to maintainers (two already fixed). That deserves credit.\n\nThe main result is clear and useful: best single decompiler (Procyon) recompiles 84% of the class files and passes original tests for 78%; a trivial 'try them in order' ensemble reaches 95%/91%. No single tool handles everything, and the ensemble gain is real.\n\nThe soft spot is the semantic metric. Definition 3 makes 'semantically equivalent modulo inputs' mean 'passes the project's test suite' on classes that are covered by at least one test. That is a pragmatic choice, but it means 78% and 91% are upper bounds on actual behavioral equivalence. The paper acknowledges the coverage threat in Section V but does not quantify test adequacy per class or bound how much the rates would drop under stronger coverage. The stress-test note is right about this. It is a limitation, not a fatal flaw: the qualitative ordering among decompilers and the no-single-perfect-tool claim would very likely survive with better tests. Just don't quote the 78% as a true equivalence rate.\n\nSmaller issues: the paper is a bit loose in places (the conclusion says 'compiled with 2 different decompilers' when it means compilers), and the public repository isn't pinned to a commit, which makes exact replication clunkier than it could be. None of this changes the conclusion.\n\nWho should read this: anyone choosing a Java decompiler in practice, and researchers working on decompiler evaluation or repair. It is a solid basis for an artifact and a benchmark. I would send it to peer review and, if I worked in this area, I would cite it.","headline":"A solid, honest empirical study of eight Java decompilers on 2041 real-world classes; the headline semantic rates are upper bounds because they mean 'passes the existing tests,' but the no-single-tool conclusion holds.","tokens_in":16262,"tokens_out":2877,"would_cite":true,"duration_ms":26616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper measures eight Java decompilers on 2,041 real-world classes and finds that no single tool produces correct, behavior-preserving source code for all of them, while a naive ensemble of decompilers outperforms every individual tool.","keywords":["Java bytecode","decompilation","equivalence modulo inputs","decompiler evaluation","reverse engineering","syntactic distortion","ensemble decompilation","empirical software engineering"],"falsifier":"Re-run the experiment's semantic-equivalence check on the classes that a decompiler passes, but replace the original test suite with coverage-guided or randomly generated inputs that exercise the differing bytecode paths; if a measurable fraction of the 78% set then fails, the reported semantic-equivalence rates are optimistic.","tokens_in":15249,"feed_emoji":"🧩","tokens_out":6744,"duration_ms":63874,"temperature":0.7,"pith_summary":"The paper measures eight Java decompilers on 2,041 classes taken from 14 real open-source projects, compiled with two different compilers, and evaluates three properties of the decompiled source: whether it recompiles, whether it behaves like the original on the project's test suite, and how far its syntax has drifted from the original. Its central finding is that no single decompiler can handle the variety of bytecode that real programs produce: the best tool, Procyon, recompiles 84% of classes and is semantically equivalent modulo inputs for 78% of test-covered classes, while a naive ensemble that tries decompilers in order recompiles 95% and passes tests for 91%. The paper also documents deceptive decompilation, where output compiles but changes behavior, and isolates 157 classes that no tested decompiler handles correctly. A sympathetic reader would care because decompilation is a routine part of reverse engineering and library inspection, and this study provides the first large-scale quantified basis for choosing among tools.","feed_headline":"No Java decompiler handles every real-world class","feed_subtitle":"Best single tool passed tests for 78% of classes; an eight-tool ensemble passed 91%.","key_machinery":"The load-bearing mechanism is equivalence modulo inputs (EMI), borrowed from compiler validation: a decompiled program is judged semantically equivalent to the original if it passes the original project's test suite. The pipeline compiles source to bytecode with javac or ecj, decompiles each class, recompiles the result, compares original and recompiled bytecode with Jardiff for strict equivalence, and runs the project's tests on the recompiled classes; syntactic distortion is measured separately as the minimum number of edits to turn the original AST into the decompiled AST. EMI converts semantic evaluation from a manual task into an automated, scalable check, which is what makes a study at this scale possible.","core_discovery":"The paper's central claim is that Java decompilation is a diverse, imperfect art: because compilation irreversibly discards information, each decompiler reconstructs the missing structure through different assumptions, and no single implementation covers the full space of real-world bytecode. Using an automated equivalence-modulo-inputs pipeline, the paper shows that the highest-scoring decompiler, Procyon, produces semantically equivalent code for 78% of test-covered classes, while a naive multi-decompiler ensemble reaches 91% because different tools succeed on disjoint sets of classes, with 276 classes handled correctly by only one of six decompilers and 157 classes by none. The paper further classifies deceptive decompilation as a distinct failure mode and reports that even the best tools exhibit it.","pith_inferences":["Because the test-suite proxy is only as strong as its coverage, the true rate of behavior-preserving decompilation is probably lower than 78%, and the paper's contribution is the measurement mechanism rather than a precise upper bound on decompiler quality.","The Multi-DC result suggests an obvious product: an ensemble decompiler that selects or merges per-method outputs could plausibly push coverage beyond 95%, since failures are concentrated in 157 classes rather than spread uniformly.","The same EMI pipeline could be transferred to other bytecode formats such as Android DEX or .NET CIL, or to obfuscated Java bytecode, where the diversity of decompiler assumptions is likely to matter even more."],"forward_implications":["Decompiler users cannot safely treat any single tool's output as authoritative: even the best tool produces behavior-changing output on some fraction of classes.","Trying multiple decompilers in a quality-ranked order is a cheap way to gain 11 points of recompilability and 13 points of test-passing coverage over the best single decompiler.","Decompiler maintainers can embed the EMI-based pipeline in their regression testing; the study already led to two fixed bugs and three acknowledged ones.","The 157 classes that no tested decompiler handles correctly constitute a concrete, reusable benchmark for future decompilation research.","Tools that minimize syntactic distortion are not the same as tools that maximize recompilability, so the right decompiler depends on whether the goal is human reading or further compilation."],"supporting_citations":[{"why":"It supplies equivalence modulo inputs, the paper's semantic-equivalence notion and the basis for the automated pipeline.","marker":"[10]"},{"why":"It provides the previous small-scale manual evaluation of Java decompilers that this study extends, along with the notion of deceptive decompilation.","marker":"[8]"},{"why":"It offers a later decompiler evaluation used as a comparison point and as a source of examples for the DcTest project.","marker":"[9]"},{"why":"It provides the Spoon library and the 13-project benchmark dataset that the study reuses for real-world classes.","marker":"[21]"},{"why":"It supplies the AST edit-distance algorithm used to compute syntactic distortion.","marker":"[11]"},{"why":"It supports the EMI idea by applying similar input-based equivalence checking to coverage tools.","marker":"[12]"}],"fun_headline_variants":["No single Java decompiler handles all classes; ensemble gets 91%","Best Java decompiler only 78% accurate; combine tools for 91%","Ensemble of 8 decompilers beats best single tool (91% vs 78%)","Java decompilation: combine eight tools for 91% accuracy vs 78% best","No Java decompiler is enough; ensemble of eight reaches 91%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire semantic-equivalence measurement assumes that passing the project's existing test suite demonstrates that decompiled code behaves like the original; when tests do not cover the constructs a decompiler changed, behavior-changing output is silently counted as equivalent.","fun_headline_variants_meta":{"raw":{"variants":["No single Java decompiler handles all classes; ensemble gets 91%","Best Java decompiler only 78% accurate; combine tools for 91%","Ensemble of 8 decompilers beats best single tool (91% vs 78%)","Java decompilation: combine eight tools for 91% accuracy vs 78% best","No Java decompiler is enough; ensemble of eight reaches 91%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000864,"raw_usage":{"total_tokens":3722,"prompt_tokens":895,"completion_tokens":2827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":511,"tokens_out":2827,"duration_ms":19401,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:31:29.723892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment's semantic-equivalence check on the classes that a decompiler passes, but replace the original test suite with coverage-guided or randomly generated inputs that exercise the differing bytecode paths; if a measurable fraction of the 78% set then fails, the reported semantic-equivalence rates are optimistic.","supporting_citations":[{"cited_title":"Compiler Validation via Equivalence Modulo Inputs,","cited_arxiv_id":null,"evidence_quote":"It supplies equivalence modulo inputs, the paper's semantic-equivalence notion and the basis for the automated pipeline."},{"cited_title":"An Evaluation of Current Java Bytecode Decompilers,","cited_arxiv_id":null,"evidence_quote":"It provides the previous small-scale manual evaluation of Java decompilers that this study extends, along with the notion of deceptive decompilation."},{"cited_title":"An Evaluation of Output from Current Java Bytecode Decompilers: Is it Android Which is Responsible for Such Quality Boost?,","cited_arxiv_id":null,"evidence_quote":"It offers a later decompiler evaluation used as a comparison point and as a source of examples for the DcTest project."},{"cited_title":"Spoon: A Library for Implementing Analyses and Transformations of Java Source Code,","cited_arxiv_id":null,"evidence_quote":"It provides the Spoon library and the 13-project benchmark dataset that the study reuses for real-world classes."},{"cited_title":"Fine-grained and Accurate Source Code Differencing,","cited_arxiv_id":null,"evidence_quote":"It supplies the AST edit-distance algorithm used to compute syntactic distortion."},{"cited_title":"Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing,","cited_arxiv_id":null,"evidence_quote":"It supports the EMI idea by applying similar input-based equivalence checking to coverage tools."}],"review_version":1}