{"id":"40729c9b-f2f3-4b17-985a-c101c0a34513","arxiv_id":"2506.06764","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CCTR is a test-aware complexity metric that adds assertion, mocking, and annotation counts to the traditional nesting score, but its claim to reflect developer perception is not yet validated.","lead":"The paper proposes CCTR, a new score for how hard a unit test is to understand: it adds up control-flow nesting, assertions, mocking calls, and annotations. The authors report that CCTR separates neat LLM-written tests from messy EvoSuite tests more clearly than the existing Cognitive Complexity metric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CCTR's core claim that it reflects developer-perceived effort is untested: equal component weights are arbitrary, and the reported discrimination across generators is confounded with suite size, so the metric's validity is not established.","rationale":"The paper has genuine strengths: a clearly defined metric, a large released dataset of 15,750 generated test suites, and a reasonable critique of SonarSource Cognitive Complexity's blind spots for test code. The illustrative examples in Section II and the aggregate observations in Section III are useful descriptive evidence. However, the paper's contribution is explicitly framed as a readability-aware metric grounded in developer perception, and that framing is what makes the central claim testable. The evaluation never measures developer perception; it only observes that CCTR produces numerically different scores for different generators. Since those scores are sums of raw counts, and since EvoSuite suites are systematically longer and contain more methods, the observed separation between EvoSuite and LLM-generated tests is explainable by suite size alone. The equal-weight assumption in Eq. (1) is made without empirical support, and the paper itself defers weight tuning and human validation to future work. This is not a technical error in the metric's definition, but it is a gap between the evidence and the claim that CCTR 'better reflects developer-perceived effort.' The Reader's verdict of REJECT, based on the overclaim and the missing ground truth, is appropriate. I would not change that verdict; the paper could be strengthened by adding a developer study or by re-scoping its claims to descriptive structural differentiation rather than perceived effort.","tokens_in":9187,"tokens_out":3313,"duration_ms":39871,"concrete_test":"Select a stratified sample of roughly 100 test methods or suites from the released dataset spanning EvoSuite, GPT-4o, and Mistral. Collect developer-perceived effort rankings (or use an existing validated human-labeled readability dataset such as Winkler et al. or Daka et al.). Compute the Spearman rank correlation between CCTR and human rankings, and compare it with the correlations for LOC, method count, and SonarSource Cognitive Complexity. If CCTR's correlation is not significantly positive, or if it is no better than the LOC/method-count baselines after controlling for those covariates, then the central claim that CCTR reflects developer-perceived effort is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CCTR 'better reflects developer-perceived effort'—is not supported by the evidence offered. Eq. (1) defines CCTR as a sum of four components with α = β = γ = δ = 1.0. The paper motivates each component separately via prior work, but supplies no empirical justification that these components contribute equally (or even additively) to perceived comprehension effort; Section V-B explicitly admits that weight tuning is future work. The main empirical result in Section V-C, Table VI, is arguably a size artifact: EvoSuite suites have roughly 2–3× more LOC and methods per suite than LLM-generated suites (Table III), and CCTR counts raw numbers of assertions, mocks, and annotations. Higher EvoSuite CCTR scores may therefore reflect longer, method-denser classes rather than validated 'fragmentation' or comprehension difficulty. The paper provides no human-study ground truth, no correlation with readability judgments, and no baseline comparison such as LOC or method count. The paper's own Limitations section and Future Work concede that 'a human study to validate the correlation between CCTR scores and developer-perceived test comprehensibility' remains to be done, while Section V-C states that the results 'support that CCTR better reflects test comprehension effort, aligning with developer-centric principles and structural variation.' That is an overclaim relative to the evidence. The load-bearing assumption is that raw counts with equal weights approximate perceived effort; currently that assumption is untested and confounded by suite size.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CCTR, a test-aware cognitive complexity metric for unit tests, defined in Eq. (1) as a weighted sum of control-flow nesting, assertions, mocking constructs, and annotations. It evaluates CCTR on 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large-1024 across Defects4J and SF110, comparing CCTR scores with Cyclomatic Complexity, SonarSource Cognitive Complexity, and Scalabrino readability scores. The paper claims that CCTR effectively discriminates between structured and fragmented test suites and \"better reflects developer-perceived effort\" (abstract, Section V-C). It also contributes a public dataset, prompts, and evaluation scripts.","tokens_in":9612,"tokens_out":4848,"duration_ms":52059,"significance":"A validated test-specific complexity metric would be a useful contribution, because existing metrics designed for functional code are known to assign near-zero scores to many generated tests. The paper has concrete strengths: it identifies a real gap, constructs a large and reproducible corpus (15,750 suites, public artifacts), and proposes a simple, interpretable metric that extends conventional cognitive complexity with test-specific constructs. However, the central claim that CCTR reflects developer-perceived effort is not supported by the evidence: the weights are asserted rather than derived or calibrated, no human ground truth is used, and the empirical comparison is descriptive and confounded with suite size. As it stands, the paper is best read as a proposal with exploratory observations, not as a validated metric.","major_comments":[{"comment":"The claim that CCTR \"better reflects developer-perceived effort\" is not supported by the evidence presented. No human study or established readability ground truth is used; Section VI explicitly lists \"a human study to validate the correlation between CCTR scores and developer-perceived test comprehensibility\" as future work. The supporting evidence in Section V-C consists of score distributions (Table VI) and author-assigned perceived-complexity labels (Tables I and VII). These do not demonstrate that CCTR aligns with developer perception.","section":"Abstract and Section V-C"},{"comment":"The equal weights α = β = γ = δ = 1.0 are introduced without empirical justification, and Section V-B admits that weight tuning is future work. Since the metric's ranking of test suites depends directly on these weights, the result is not a measured property of test comprehension but an arbitrary design choice. A sensitivity analysis over weight combinations, or calibration against human judgments, is needed before the metric's scores can be interpreted as reflecting comprehension effort.","section":"Section V-B, Eq. (1)"},{"comment":"The reported discrimination between EvoSuite and LLM-generated suites is confounded with suite size. According to Table III, EvoSuite suites have roughly 2–3× more LOC and methods per suite than LLM-generated suites, and CCTR sums raw counts of assertions, mocks, and annotations. Higher EvoSuite CCTR scores (Table VI) may therefore simply reflect longer, method-denser classes rather than the \"fragmented structure\" the paper attributes to them. The paper does not compare CCTR against LOC, method count, or other size baselines, nor does it normalize or control for size.","section":"Section V-C, Tables III and VI"},{"comment":"The treatment of non-compiling LLM-generated tests is not described precisely. Table III reports compilation rates of 100% for EvoSuite, 67.58% for GPT-4o, and 42.42% for Mistral Large-1024, but the paper does not state whether complexity and readability metrics were computed on all generated suites or only on those that compile. If non-compiling tests were included, PMD and CCTR scores may be unreliable; if they were excluded, the comparison is not based on the same population. This must be clarified and justified.","section":"Section III (A-B)"}],"minor_comments":[{"comment":"There is a typo in the introduction: \"congitive effort\" should be \"cognitive effort.\"","section":"Section I"},{"comment":"The metric is called \"Test-Aware Cognitive Complexity\" in the abstract but \"Cognitive Complexity for Test Readability\" in Section V; the naming should be made consistent.","section":"Abstract and Section V"},{"comment":"The Mistral-L rows for Defects4J and SF110 are identical (LOC 112.06, tokens 939.33, methods 16.20, compilability 42.42%). Please verify whether these values are correct or a copy-paste error.","section":"Table III"},{"comment":"The \"Perceived Complexity\" labels are assigned by the authors and are used as if they were ground truth. They should be clearly presented as informal illustrations rather than empirical measurements.","section":"Tables I and VII"}],"recommendation":"reject","confidential_remarks":"The central contribution is a proposed metric, but the paper's title and abstract promise empirical grounding in developer perception that the body does not deliver. The main problems—missing human validation, arbitrary weights, and the size confound—are load-bearing and would require a substantial new empirical study rather than local corrections. I therefore recommend rejection, though a substantially revised version that reframes the contribution as an exploratory metric proposal and adds at least a size-controlled and sensitivity-analyzed evaluation could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the paper defines CCTR, a simple additive metric for unit test complexity that counts nesting plus assertions, mocks, and annotations, reports it over 15,750 generated suites from EvoSuite, GPT-4o, and Mistral, and releases all data and scripts. That's a useful artifact, and the motivating observation is real: Sonar's Cognitive Complexity gives near-zero scores to most LLM-generated tests and fails to distinguish them from more fragmented EvoSuite tests. The metric itself is easy to understand and compute, and the authors are transparent about its limits in Section VI.\n\nWhat is actually new: the combination of test-specific constructs into a single weighted sum is new relative to the literature they cite. It is a modest extension, not a deep framework, and they don't claim more than that. The release of the full dataset and evaluation scripts is a genuine plus; reproducibility is handled properly. The citation pattern is fine; the prior work they draw on is real, and the self-citations are to their own earlier test-generation studies, which is appropriate.\n\nNow the soft spots. The core claim, in the abstract and in Section V-C, is that CCTR 'better reflects developer-perceived effort' and 'aligns with developer-centric principles.' As the paper's own Future Work admits, there is no human study. The weights are set to 1.0 without empirical derivation, and the authors call tuning future work. So the central validity claim is untested. I also think the empirical contrast in Table VI is partly a size artifact: EvoSuite suites have roughly two to three times the methods and LOC (Table III), and CCTR counts raw numbers of assertions, mocks, and annotations. Higher EvoSuite scores may just be longer classes, not more complex ones. The paper does not control for suite size or method count, and it does not compare against a baseline like LOC in predicting anything.\n\nThat said, I don't think these are fatal flaws in the sense of the metric being incoherent. They are missing validation and an overclaim relative to the evidence. If the authors weaken the language to 'CCTR captures structural variation in test constructs and is a candidate heuristic for test complexity,' the paper becomes a reasonable tool paper. As written, the abstract overpromises.\n\nMy take: send it to peer review, but expect a major-revision round. The artifact and the problem are real, and the measurement gap is worth attention. A serious referee can push for either a human study or a much more cautious conclusion. The data release makes that feasible.","headline":"A test-specific complexity metric that is a good idea and a solid artifact, but the central claim that it reflects developer-perceived effort is unvalidated and the abstract overstates the evidence.","tokens_in":10027,"tokens_out":2348,"would_cite":false,"duration_ms":25255,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"New metric CCTR tells structured test suites from fragmented ones, filling a blind spot in existing complexity measures.","keywords":["cognitive complexity","unit testing","test readability","CCTR","LLM-generated tests","EvoSuite","software metrics","test comprehension"],"falsifier":"Run a controlled study where developers rate the comprehension effort of test suites that are matched for CCTR score but differ in composition (e.g., assertion-heavy versus mock-heavy), or suites differing in CCTR but rated equally difficult. If the ratings do not track CCTR in either direction, the metric's claim to reflect developer-perceived effort fails. A concrete check: take two 12-method suites, one with 30 assertions and no mocks and one with 10 mocks and 5 assertions, both scoring the same CCTR total, and ask developers which is harder; any systematic preference breaks the equal-weight assumption.","tokens_in":9001,"feed_emoji":"🧪","tokens_out":6746,"duration_ms":63773,"temperature":0.7,"pith_summary":"Most complexity metrics used to judge automatically generated unit tests were built for production code, not test code, and the paper demonstrates the mismatch: SonarSource's Cognitive Complexity assigns zero to over 99% of LLM-generated test methods while also scoring simple nested dummy code higher than real test suites. The paper proposes CCTR, a test-aware metric that adds three test-specific signals—assertion density, mocking constructs, and annotation roles—to the traditional nesting-based score. Across 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large over 350 Java classes, CCTR produces clearly separated scores for structured LLM tests and fragmented EvoSuite tests. The aim is to provide a readability-aware complexity score that can be used to evaluate, refactor, and improve generated tests.","feed_headline":"New metric CCTR tells structured test suites from fragmented ones","feed_subtitle":"It counts assertions, mocks, and annotations, where SonarSource's cognitive complexity sees zero.","key_machinery":"The central object is the CCTR formula, a weighted sum that treats four signals as equally important proxies for the mental effort of reading a test method: nesting complexity N, assertion count A, mocking count M, and annotation signaling T (with specialized annotations weighted double). This identity carries the argument because it converts the missing dimensions of test readability into a single comparable number, and the equal-weight choice (α = β = γ = δ = 1.0) is what keeps the metric lightweight and interpretable while producing the score separations reported in the evaluation.","core_discovery":"SonarSource's Cognitive Complexity, which penalizes nested control flow and logical operators, was designed for functional code and ignores the constructs that dominate unit tests. The paper claims that this makes it blind to the actual structure of test code: an LLM test suite with meaningful method names, assertions, and annotations scores 0, while a deeply nested dummy loop scores 12. CCTR is defined as CCTR = α·N + β·A + γ·M + δ·T, with all weights initially set to 1.0, where N is the original control-flow nesting complexity, A counts assertions and fail() statements, M counts mocking calls such as mock(), verify(), and when(), and T adds +1 per common annotation (@Test, @BeforeEach, @AfterEach) and +2 for @ParameterizedTest. On 15,750 suites, CCTR assigns mean scores of 26–30 to GPT-4o and Mistral tests and 39–59 to EvoSuite tests, separating generation styles that SonarSource's metric collapses. The paper concludes that CCTR better reflects test comprehension effort and structural variation in test code.","pith_inferences":["The equal weights are an uncalibrated default; a developer study could find that one assertion and one mock call do not cost the same mental effort, in which case the ranking of generators would change even though the formula stays the same.","Because CCTR is purely syntactic, it can be gamed: merging assertions into a single helper call or deleting annotations would lower the score without necessarily improving readability, so a stress test under semantics-preserving transformations would show how stable the metric is.","The near-zero SonarSource scores for LLM tests may reflect that LLMs write flat, non-nested tests; CCTR's nonzero values then partly measure verbosity and annotation load, so the metric's 'effort' interpretation needs validation against actual comprehension times rather than structural intuition alone.","A natural next test is to check whether CCTR correlates with test usefulness, for example whether suites with very high CCTR (EvoSuite) also show lower fault-detection efficiency, linking readability scoring to test effectiveness."],"forward_implications":["CCTR assigns nonzero scores to test suites that SonarSource's Cognitive Complexity rates as zero, so it can serve as a discriminating signal in automated review of generated tests.","Because CCTR scales to large batches, it enables comparison of test synthesis quality across tools and models where existing readability models or complexity metrics give little information.","The score differences between EvoSuite and LLM-generated suites quantify fragmentation and semantic density, giving a concrete target for test refactoring and generation prompts.","CCTR is intended to complement dynamic analyses: it covers structural and semantic effort, while coverage and fault-detection metrics cover behavioral effectiveness."],"supporting_citations":[{"why":"Defines the SonarSource Cognitive Complexity specification that CCTR extends and critiques.","marker":"[10]"},{"why":"Provides the developer-centric test comprehension dimensions (structure, naming, logic/purpose) that ground CCTR's components.","marker":"[13]"},{"why":"Supplies the 104-feature readability model used to compare LLM and EvoSuite suites on perceived readability.","marker":"[15]"},{"why":"Validates the Scalabrino readability model against human ratings, justifying its use as the comparison baseline.","marker":"[16]"},{"why":"Defines Cyclomatic Complexity, the control-flow baseline compared in the evaluation.","marker":"[9]"},{"why":"Supplies Defects4J, one of the two Java corpora used for test generation and evaluation.","marker":"[17]"},{"why":"Provides the DynaMOSA search algorithm and the SF110 corpus used to generate and benchmark tests.","marker":"[18]"},{"why":"Shows that annotations affect Java code readability, supporting CCTR's annotation signaling term.","marker":"[26]"},{"why":"Identifies EvoSuite as the search-based tool whose generated suites serve as the fragmented comparison baseline.","marker":"[1]"}],"fun_headline_variants":["CCTR metric reveals structure SonarSource's complexity misses","Test-aware metric CCTR counts assertions and mocks, not just nesting","CCTR separates structured from fragmented unit tests, unlike SonarSource","Readability-aware metric for unit tests sees what cognitive complexity ignores","CCTR scores test suites by assertions, mocks, and annotations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that counting nesting, assertions, mocks, and annotations with equal weight captures how hard developers actually find a test to read; if developers weight these factors differently, the metric's rankings may not match perceived effort.","fun_headline_variants_meta":{"raw":{"variants":["CCTR metric reveals structure SonarSource's complexity misses","Test-aware metric CCTR counts assertions and mocks, not just nesting","CCTR separates structured from fragmented unit tests, unlike SonarSource","Readability-aware metric for unit tests sees what cognitive complexity ignores","CCTR scores test suites by assertions, mocks, and annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1318,"prompt_tokens":1001,"completion_tokens":317,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":617,"tokens_out":317,"duration_ms":3752,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:49:09.501722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled study where developers rate the comprehension effort of test suites that are matched for CCTR score but differ in composition (e.g., assertion-heavy versus mock-heavy), or suites differing in CCTR but rated equally difficult. If the ratings do not track CCTR in either direction, the metric's claim to reflect developer-perceived effort fails. A concrete check: take two 12-method suites, one with 30 assertions and no mocks and one with 10 mocks and 5 assertions, both scoring the same CCTR total, and ask developers which is harder; any systematic preference breaks the equal-weight assumption.","supporting_citations":[{"cited_title":"Cognitive complexity: A new way of measuring under- standability,","cited_arxiv_id":null,"evidence_quote":"Defines the SonarSource Cognitive Complexity specification that CCTR extends and critiques."},{"cited_title":"Investigating the readability of test code,","cited_arxiv_id":null,"evidence_quote":"Provides the developer-centric test comprehension dimensions (structure, naming, logic/purpose) that ground CCTR's components."},{"cited_title":"A comprehensive model for code readability,","cited_arxiv_id":null,"evidence_quote":"Supplies the 104-feature readability model used to compare LLM and EvoSuite suites on perceived readability."},{"cited_title":"Reassessing java code readability models with a human- centered approach,","cited_arxiv_id":null,"evidence_quote":"Validates the Scalabrino readability model against human ratings, justifying its use as the comparison baseline."},{"cited_title":"A complexity measure,","cited_arxiv_id":null,"evidence_quote":"Defines Cyclomatic Complexity, the control-flow baseline compared in the evaluation."},{"cited_title":"Defects4j: A database of existing faults to enable controlled testing studies for java programs,","cited_arxiv_id":null,"evidence_quote":"Supplies Defects4J, one of the two Java corpora used for test generation and evaluation."},{"cited_title":"Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,","cited_arxiv_id":null,"evidence_quote":"Provides the DynaMOSA search algorithm and the SF110 corpus used to generate and benchmark tests."},{"cited_title":"How do annotations affect java code readability?","cited_arxiv_id":null,"evidence_quote":"Shows that annotations affect Java code readability, supporting CCTR's annotation signaling term."},{"cited_title":"Evosuite: automatic test suite generation for object-oriented software,","cited_arxiv_id":null,"evidence_quote":"Identifies EvoSuite as the search-based tool whose generated suites serve as the fragmented comparison baseline."}],"review_version":1}