{"id":"1da86a56-dc46-47b8-ade1-22102090c821","arxiv_id":"2501.12680","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reordering tests in 81 Jest projects uncovered 55 order-dependent tests, with shared mocking state reported as a newly identified cause.","lead":"This paper presents a randomized test-reordering tool, JS-TOD, applied to 81 popular JavaScript projects using Jest, and reports 55 order-dependent flaky tests caused by shared files or shared mocking state. A smart generalist might read it to see how hidden test-order dependencies can make JavaScript test suites pass or fail unpredictably.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control baseline for AST-reordered files: failures in reordered test files were not checked against a transformed original-order control, so transformation artifacts (e.g., Jest hoisting of jest.mock) may inflate the reported order-dependent counts.","rationale":"The paper makes a credible empirical contribution: the manual root-cause analysis is careful, the examples are illustrative, and the replication package is provided. The main weakness is the detection pipeline's lack of a baseline control for the transformation itself. The reader identified the same core issue (semantic equivalence of the Babel rewrite). I sharpen it by pointing to Jest's hoisting of `jest.mock` as a specific mechanism by which the transformation can create failures even when the code is syntactically faithfully reconstructed. Because the paper's own Section V-B1 describes a shared-mocking-state example whose explanation conflicts with hoisting, the concern is not hypothetical. Nevertheless, the presence of clearly genuine order-dependent tests (call-count assertions) means the central claim that order-dependence exists in Jest projects is supported; only the exact counts and the novelty of 'shared mocking state' need refinement. The conditional verdict is appropriate.","tokens_in":17527,"tokens_out":11989,"duration_ms":113076,"concrete_test":"For each project with detected OD tests, use JS-TOD to generate a transformed test file containing the same extracted tests/describe blocks in their original order (a control). Run each control file 10 times. If any of the 55 tests fails in the control, that failure is a transformation artifact, not order-dependence. In addition, for the shared-mocking-state cases, run the original test files with Jest's execution-order randomization (e.g., `--randomize` or a custom sequencer) without changing source order; if the tests do not fail under execution-order randomization, they are source-order artifacts caused by `jest.mock` hoisting, not execution-order flakiness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"JS-TOD detects order-dependence by AST-extracting tests and describe blocks and saving them as new test files in a reordered sequence (Section III-B). The central premise is that any new failure is caused by the new order, not by the transformation. The paper reports nine projects where the transformation dropped code and caused failures (Section V-B1), and after manual fixes these projects passed. However, the paper never reports a control: running the transformed test file with the tests in the *original* order to confirm the transformation itself is harmless. Without this control, a test that fails in a reordered file could be failing because the transformation corrupted the file, not because of order-dependence. This concern is concrete. In Jest, `jest.mock` calls are hoisted to the top of the file, so the mock implementation that takes effect is determined by the last such call in source order, not by execution order. Reordering test blocks physically changes the source order of `jest.mock` calls, which can change which mock factory is active for all tests in the file. The paper's shared-mocking-state example in Section V-B1 (native-testing-library) claims that a test containing a local `jest.mock('react-test-renderer', ...)`, when moved earlier, causes that mock to affect other tests. But because of hoisting, that mock should already affect all tests in the original order. The explanation is inconsistent with Jest's hoisting behavior, indicating that at least some of the 42 reported shared-mocking-state cases may be artifacts of source reordering rather than execution-order dependence. The same control deficiency applies to the 13 shared-file cases: a transformed file with missing or altered setup code could produce file-related failures regardless of order. The existence of at least some genuine order-dependent tests (e.g., the call-count assertions in jest-webextension-mock) is credible, so the central claim is not refuted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents JS-TOD, a tool that randomizes Jest tests at the level of individual tests, describe blocks, and test suites, and reruns the reordered files multiple times to detect order-dependent flaky tests. The authors evaluate the approach on 81 GitHub projects and report 55 order-dependent tests across 10 projects, attributing them to shared files (13 tests) and shared mocking state (42 tests). They claim that shared mocking state is a previously unreported cause of order-dependent flakiness. The study includes a manual root-cause analysis with independent author verification and a public replication package.","tokens_in":17784,"tokens_out":6394,"duration_ms":61918,"significance":"If the findings hold, the paper provides the first systematic evidence of order-dependent flakiness in Jest projects at scale, and its proposed new cause category (shared mocking state) would be a useful addition to the flaky-test taxonomy. The strengths are the careful two-author manual analysis with conflict resolution, the consistent failure of all 55 tests across reruns of the same order, and the availability of a replication package. The main weakness is the lack of a control for transformation equivalence, which is load-bearing for the central claim that the observed failures are caused by test order rather than by the AST rewriting itself.","major_comments":[{"comment":"The paper does not report a control in which the AST-transformed test file is run with the tests in their original order. Since the classification of a test as order-dependent depends on failures appearing only after reordering, and since the authors themselves report nine projects where the transformation dropped code (Section V-B1), the absence of this control leaves open the possibility that some of the 55 reported failures are transformation artifacts rather than order dependence.","section":"Section III-B, Algorithm 2"},{"comment":"The code-generation step is underspecified: it is unclear whether top-level statements outside test/describe nodes (imports, requires, beforeEach/afterEach hooks, and top-level jest.mock calls) are preserved in the generated files. The native-testing-library example in Listing 6 includes a top-level jest.mock and beforeEach/afterEach; if those are not included in the new test files, the observed inter-test effects would not correspond to the original test suite's semantics. The paper should state exactly which AST nodes are copied into each generated file and demonstrate that the transformed original-order file is equivalent.","section":"Section III-B, Algorithm 2"},{"comment":"The assertion that shared mocking state is a cause \"not reported previously\" is not supported by a comparison with prior classification taxonomies. In Java, for example, mocking frameworks maintain global static state, and prior taxonomies (e.g., Luo et al. [11]) subsume such state under shared static state. The paper should either differentiate shared mocking state from existing categories with concrete evidence from prior work or soften the novelty claim.","section":"Section V-B1 and abstract"},{"comment":"The statement \"All 55 tests fail across all 10 reruns of the same order, confirming that they indeed are order-dependent tests\" overstates the evidence. Consistent failure across reruns of the same reordered file only confirms deterministic behavior of that transformed file; it does not by itself confirm order dependence without a comparison against the transformed original-order file. This wording should be revised to reflect the need for the control.","section":"Section V-A"}],"minor_comments":[{"comment":"The footnote says shaded rows are projects with failing tests but not order-dependent flaky tests; for clarity, consider explicitly distinguishing these non-OD failures from the OD rows in the table header or caption.","section":"Table III"},{"comment":"The \"Total\" row sums the three reordering levels, but the levels overlap in projects; reporting unique project counts in addition to the sums would avoid conflation.","section":"Table II"},{"comment":"Reference [39] contains a typo: \"Ssoftware\" should be \"Software\".","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible but rests on the assumption that the AST rewriting is semantics-preserving. The authors should be asked to provide the original-order control as a condition of publication. The novelty claim about shared mocking state also deserves scrutiny in light of prior static-state categories."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this paper does real work: it actively reorders Jest tests, describe blocks, and suites across 81 popular GitHub projects, finds 55 order-dependent tests, and proposes a new cause category—shared mocking state—that the cited Java/Python literature does not report. Second, the methodology has a load-bearing gap: JS-TOD rewrites test files via Babel but never runs a transformed file in the original order as a control. Section V-B1 admits nine projects where the rewrite dropped code and had to be manually fixed. Without a control, some of the 55 reported failures could be transformation artifacts rather than order dependence. The new-cause claim also has a concrete problem. The native-testing-library example in Listing 6 says that moving a test containing a local jest.mock('react-test-renderer', ...) earlier in the file causes that mock to pollute other tests. But Jest hoists jest.mock calls to the top of the module scope, so that mock should already be active in the original order. The described mechanism is inconsistent with Jest semantics, and some of the 42 shared-mocking-state cases may be artifacts of physical source reordering. The other large cluster—35 call-count assertions in jest-webextension-mock—is much cleaner and almost certainly genuine order dependence. What the paper does well: the manual root-cause analysis is careful (two independent inspectors, a third for disagreements), all 55 tests failed consistently across reruns of the same order, and the tool and replication package are public. The three-level reordering—individual tests, describe blocks, and suites—is a real extension beyond iDFlakies and FlaPy, even if the core idea is ported. The authors are upfront about threats: only 10 reorders, no recall measurement, manual project selection. They also report, rather than hide, the transformation failures; that honesty is creditworthy. The missing control baseline is the main issue, and it directly affects the headline count. The 'not reported previously' claim lacks a systematic literature comparison. The prevalence numbers are lower bounds given the sampling and the all-passing-default-order filter. None of this refutes the existence of order-dependent flakiness in Jest—the paper has that—but the shared-mocking-state category needs re-verification with a proper original-order control of the transformed files, and the Listing 6 explanation should be reconciled with Jest's hoisting behavior. I would bring this to a reading group and likely cite the paper. A serious editor should send it to peer review: it needs revision, not rejection. The referee should push for the control experiment and a re-classification of the shared-mocking-state examples.","headline":"First systematic Jest order-dependence study with a plausible new cause category, but the missing transformed-original-order control and a likely hoisting artifact in the headline example undermine the shared-mocking-state count.","tokens_in":884,"tokens_out":938,"would_cite":true,"duration_ms":30171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Systematic reordering in 81 Jest projects finds 55 order-dependent flaky tests, most caused by shared mocking state.","keywords":["flaky tests","order-dependent tests","Jest","JavaScript testing","shared mocking state","shared files","test reordering","test state pollution"],"falsifier":"Rerun the reported 55 tests in both their failing and passing orders with the transformation removed, using Jest's own randomize option on the original files; if a reported test fails in both orders, or passes in the order that supposedly fails, its order-dependence classification is wrong. Alternatively, add a global beforeEach with jest.clearAllMocks() to jest-webextension-mock and check whether all 35 reported failures disappear.","tokens_in":17321,"feed_emoji":"🔀","tokens_out":5849,"duration_ms":56193,"temperature":0.7,"pith_summary":"Flaky tests are tests whose outcomes vary between runs, and one known trigger is the order in which tests execute. This paper asks whether JavaScript projects using Jest, the most widely used JavaScript testing framework, contain hidden order-dependent tests and what causes them. To find out, the authors built a tool that reorders tests, describe blocks, and whole test suites into random orders and reruns each order, then manually analysed every failure. Across 81 projects they found 55 order-dependent tests: 52 between individual tests, 3 between describe blocks, and none between test suites. The causes were shared files (13) and shared mocking state (42), with mocking-state persistence a cause that earlier flaky-test studies in Java and Python had not reported.","feed_headline":"Reordering Jest tests exposes 55 order-dependent flakes","feed_subtitle":"Most trace to shared mocking state, a cause previous flaky-test studies did not report.","key_machinery":"The load-bearing mechanism is JS-TOD, the paper's detector, built on four pieces: Jest's -listTests option to enumerate suites, a custom TestSequencer subclass that enforces a supplied suite order, Babel's abstract syntax tree to lift individual tests and describe blocks out of a suite into new files, and Algorithm 1, which generates up to 10 unique random orders per level (or all permutations if fewer exist). Each reordered file is rerun 10 times; a failure that recurs in every rerun of the reordered file, while passing in the default order, is classified as order-dependent. Manual inspection then classifies each failure as stemming from shared files or shared mocking state.","core_discovery":"The paper's central claim is that Jest projects do contain real order-dependent flakiness, and that the dominant mechanism is shared mocking state rather than the shared-file or static-state mechanisms documented for Java and Python. A test is counted as order-dependent when it passes in the project's default order, fails in every rerun of a reordered file, and the failure disappears when the original order is restored. Of the 55 detected tests, 52 appear when the order of tests inside a suite changes, and 3 appear when the order of describe blocks changes; no order dependence was found between test suites in the 49 projects analysed at that level. Manual inspection attributes 13 tests to shared files and 42 to mocking state that persists across tests through Jest's module and mock registry, a category the authors say previous order-dependence studies did not report.","pith_inferences":["If the paper's central claim holds, its count of 55 is a lower bound: with only 10 random orders per level, other order-dependent tests may require rarer orderings, and more aggressive or pairwise reordering would likely find more.","The new shared-mocking-state cause suggests a cheap preventive check: statically flag any test that calls jest.mock without a matching reset in beforeEach or afterEach, since such mocks persist across tests.","If shared mocking state is as prevalent in other JavaScript frameworks as it is in Jest, current flaky-test taxonomies, built mostly from Java and Python, may need a new category for framework-level mock registries.","The methodology could be extended to other Jest-like frameworks by replacing the AST extraction and sequencer hooks, but the describe-block level is Jest-specific and would not transfer directly."],"forward_implications":["Developers using Jest can no longer assume that running tests in file order guarantees deterministic outcomes; hidden order dependence can surface whenever Jest reorders suites or a developer uses the randomize option.","Resetting shared state, for example jest.clearAllMocks() in a beforeEach hook, should fix a large share of order-dependent tests: the paper reports clearing mock state fixed 34 of the 39 test-level shared-mocking-state failures.","Order dependence can also hide between describe blocks, so tools and best practices should treat describe blocks as a reorderable unit, not just individual tests.","Because no order-dependent test was found at the test-suite level across 49 projects, the practical risk in Jest projects is concentrated inside suites rather than across suite files."],"supporting_citations":[{"why":"Earlier Java study establishing order dependency as a major flaky-test cause; supplies the comparative baseline the paper extends to JavaScript.","marker":"[11]"},{"why":"Earlier Python study reporting order dependency as a main flakiness cause; another baseline for prevalence comparison.","marker":"[12]"},{"why":"Prior JavaScript flaky-test study from issue trackers finding few order-dependent cases; motivates active detection rather than relying on reported flakes.","marker":"[13]"},{"why":"Jest community feature request for test randomization; motivates the tool's reordering approach.","marker":"[17]"},{"why":"Defines the victim, polluter, and brittle classification used to interpret the detected failures.","marker":"[19]"},{"why":"Babel toolchain used to parse suites and extract describe blocks and tests for reordering.","marker":"[28]"},{"why":"Previous work detecting order-dependent tests in Java via randomized orders and classifying them; supplies the methodological template.","marker":"[39]"}],"fun_headline_variants":["Jest order shuffling uncovers 55 flaky tests","Shared mocking state drives most Jest order-flakes","New flaky-test cause: Jest sharing mocking state","55 Jest tests fail only on reorder, mocking blamed","Most Jest flakes stem from shared mock state"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole classification rests on the assumption that Babel's AST-based rewriting of a test suite into reordered files leaves the tests' behavior unchanged, so a new failure reflects order rather than transformation; the authors themselves report nine projects where Babel missed code and required manual repair.","fun_headline_variants_meta":{"raw":{"variants":["Jest order shuffling uncovers 55 flaky tests","Shared mocking state drives most Jest order-flakes","New flaky-test cause: Jest sharing mocking state","55 Jest tests fail only on reorder, mocking blamed","Most Jest flakes stem from shared mock state"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1423,"prompt_tokens":949,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":565,"tokens_out":474,"duration_ms":5128,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:53:51.330917+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the reported 55 tests in both their failing and passing orders with the transformation removed, using Jest's own randomize option on the original files; if a reported test fails in both orders, or passes in the order that supposedly fails, its order-dependence classification is wrong. Alternatively, add a global beforeEach with jest.clearAllMocks() to jest-webextension-mock and check whether all 35 reported failures disappear.","supporting_citations":[{"cited_title":"An empirical analysis of flaky tests,","cited_arxiv_id":null,"evidence_quote":"Earlier Java study establishing order dependency as a major flaky-test cause; supplies the comparative baseline the paper extends to JavaScript."},{"cited_title":"An empirical study of flaky tests in python,","cited_arxiv_id":null,"evidence_quote":"Earlier Python study reporting order dependency as a main flakiness cause; another baseline for prevalence comparison."},{"cited_title":"An empirical study of flaky tests in javascript,","cited_arxiv_id":null,"evidence_quote":"Prior JavaScript flaky-test study from issue trackers finding few order-dependent cases; motivates active detection rather than relying on reported flakes."},{"cited_title":"Issue #4386: Flakiness in tests with Jest,","cited_arxiv_id":null,"evidence_quote":"Jest community feature request for test randomization; motivates the tool's reordering approach."},{"cited_title":"iFixFlakies: A framework for automatically fixing order-dependent flaky tests,","cited_arxiv_id":null,"evidence_quote":"Defines the victim, polluter, and brittle classification used to interpret the detected failures."},{"cited_title":"Babel · babel,","cited_arxiv_id":null,"evidence_quote":"Babel toolchain used to parse suites and extract describe blocks and tests for reordering."},{"cited_title":"iDFlakies: A framework for detecting and partially classifying flaky tests,","cited_arxiv_id":null,"evidence_quote":"Previous work detecting order-dependent tests in Java via randomized orders and classifying them; supplies the methodological template."}],"review_version":1}