REVIEW 4 major objections 4 minor 1 cited by
How Developers Choose Debugging Strategies for Challenging Web Application Defects
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Expert debugging strategy choice is driven by interacting contextual factors, with hypothesis testing as the baseline.
desk verdict Useful taxonomy of contextual factors for debugging strategy choice, but the Figure 2 decision model is a prudent reconstruction from primed self-reports, not a validated model. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the qualitative decision model in Figure 2, a factor-to-strategy mapping that shows how developers branch on checks such as whether a clear error message exists, whether the defect is reproducible, whether it is client-side, whether the developer has code access and codebase familiarity, whether the defect is user-specific or sporadic, and whether the codebase is small, deprecated, or familiar. The model is built by causal coding of interview transcripts, which locates and extracts the causal beliefs developers state between contextual factors and strategy choices. This machinery converts open-ended interviews into a compact branching structure that explains both initial strategy selection and mid-debugging switches.
What would settle it
An observational study that records developers' actual debugging sessions and finds their strategy choices are unrelated to the defect and codebase factors listed here—for instance, the same habitual strategy is used across clear and unclear, familiar and unfamiliar contexts—would falsify the claimed factor-strategy links.
Extended reading notes
Core claim
The paper's central discovery is that expert developers choose debugging strategies by weighing combinations of contextual factors, not by applying a single preferred method. Defect characteristics—clarity, reproducibility, and the root-related category covering network, data, configuration, and hardware—and codebase characteristics—familiarity, access, maintenance, technical stack, testability, and complexity—are the dominant influences, with organizational context, tool availability, individual traits, and project requirements also playing a role. Developers typically begin with hypothesis-test debugging and switch strategies as new information emerges; the proposed decision model in Figure 2 maps these factor checks to strategies such as backward-reasoning, forward-reasoning, simplification, binary-search, error-message debugging, and system-level checks. The paper also finds that experienced developers draw on strategies not previously documented and treat debugging as an occasion to improve code quality.
Load-bearing premise
The paper's conclusions rest on retrospective self-reports from 35 survey respondents and 16 interviewees accurately capturing the reasoning that actually drove their debugging choices.
Editorial extensions
If this is right
- Debugging education should teach context-assessment skills—evaluating reproducibility, clarity, codebase familiarity, and access—rather than only demonstrating individual strategies.
- Debugging tools should be designed around problem contexts, because a tool that supports one strategy, such as backward-reasoning via a slicer, may be useless in contexts that call for simplification or binary-search.
- The documented set of debugging strategies should be expanded beyond the six inherited from prior work to include system-level checks, external-resource consultation, and historical analysis with version-control bisection.
- Because codebase familiarity and access so often decide whether hypothesis-testing and backward-reasoning are viable, lowering those barriers, through better code comprehension support, should change which strategies developers can effectively use.
- Debugging is also a code-quality activity, so evaluations of debugging success should include maintainability outcomes, not only time-to-fix.
Reading between the lines
- An implication the paper leaves implicit: if strategy selection is a context-reading skill, then an IDE feature that prompts developers to check factors like reproducibility, clear error message, and codebase familiarity before choosing a tactic could serve as both a teaching tool and a test of the model.
- Beyond the paper's data, the same factor-strategy links could be tested in real time by instrumenting debugging sessions and comparing observed factor states against the Figure 2 branches, rather than relying on retrospective accounts.
- A further extension: the growing use of AI code assistants may alter the familiarity factor, since developers can quickly learn unfamiliar code; whether that shifts their strategy choices away from forward-reasoning toward hypothesis-testing is a concrete empirical question.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates how developers choose debugging strategies for challenging web application defects through two complementary studies: a survey of 35 developers with varying expertise and semi-structured interviews with 16 expert developers. It identifies six categories of contextual factors, maps them to debugging strategies in Figure 2, and claims that strategy selection is driven by interacting combinations of factors that evolve during debugging, with hypothesis testing as a baseline and code familiarity/experience as decisive influences. The paper also derives implications for debugging tools and education.
Significance. If the central claims hold, the paper makes a useful contribution by moving beyond listing debugging strategies to modeling the contextual conditions under which developers select and switch strategies, with concrete implications for tool design and debugging education. The paper's strengths include its two-study design, systematic two-round coding with multiple coders, rich participant quotes, and a candid threats-to-validity section. However, the central empirical claim requires more direct evidence that the reported factor-strategy links are not artifacts of the elicitation protocol, and the decision model in Figure 2 would need stronger empirical grounding before it can support the paper's strongest conclusions.
major comments (4)
- [Section 3.2.3] The interview protocol gave participants the six strategies in Table 3 and the four contextual factor categories before asking them to choose strategies and describe the factors they considered. This scaffolding may have elicited exactly the kind of multi-factor, strategy-switching reasoning that the paper reports as its central finding. The manuscript does not report whether the same factor-strategy links appear in unprompted narratives, such as the open-ended accounts in Study 1 or the parts of Study 2 in which participants described recent experiences in their own words. Please provide such an analysis, or explicitly reframe the central claim as 'when prompted with these categories, developers report that...'.
- [Figure 2] Figure 2 is labeled a 'Model of developer decision-making in web debugging strategy selection,' but no per-edge counts, inter-rater reliability values, or validation against observed debugging behavior are reported. The figure should either be supported with the number of participants whose transcripts support each edge and representative quotes, or relabeled as an illustrative synthesis rather than a validated decision model. In addition, several branches and edge labels (e.g., 'Deprecated?' with labels l and k) are unclear without a more detailed legend.
- [Abstract and Section 3] The abstract frames the contribution as covering 'different expertise levels,' and RQ1 asks about factors influencing strategy choice, but the Results section does not systematically compare strategy choices by expertise: Study 2 recruited only experts (8-38 years of experience) and Study 1 included many students with a median of 2 years. Claims such as 'experience and familiarity with the code are keys to making the correct decision' should be presented as participants' stated beliefs rather than as a demonstrated expertise effect.
- [Section 5] The threats-to-validity section acknowledges recall bias, sample bias, and potential codebook influence, but these admitted limitations are not connected to the strength of the central claim that real debugging decisions are driven by interacting contextual factors. Because the model is derived from retrospective self-reports, the paper should state explicitly that the model represents reported reasoning and should describe how the mitigating evidence mentioned in Section 5 (e.g., references to GitHub commits or chat history) was used in the analysis.
minor comments (4)
- [Figure 2] The text 'Need to talk to stockholders or manager' should read 'stakeholders,' and the edge labels in the figure are referenced inconsistently in the text (e.g., 'Fig 2-i, 8' and 'Fig 2-d-k-NO').
- [Tables 5-7] The asterisk and underline notation for marking the provenance of factors is not consistently followed: some underlined factors, such as 'Complexity' in Table 5, are not discussed in the corresponding text, making the notation harder to follow.
- [Section 5] The phrase 'double-blind analysis' is misleading given the collaborative open-coding process described in Section 3.2.4; please rephrase to describe the actual analysis procedure.
- [Section 2.3] The related work discussion would benefit from a closer comparison to Spinellis's bottom-up/top-down strategy-selection advice [47], since the paper's decision model partly overlaps with that distinction.
Circularity Check
Partial circularity: Study 2 supplies participants with the factor categories it later reports; open coding limits the reduction.
-
other
[Section 3.2.3 (Procedure) and Section 5 (Threats to validity)]
"The participants were provided with examples, a list of common debugging strategies (Table 3), and the four categories of contextual factors generated in Section 3.2.1. Participants were asked to choose strategies and describe the factors considered in choosing a debugging strategy, how each factor influenced their choice, and which strategies would not be effective for their specific case."
The four contextual-factor categories supplied to interviewees are the same categories that later appear in the results (Tables 5-7) and in the decision model (Figure 2). Asking participants to 'describe the factors considered' after handing them those categories means the factor-strategy links are partly elicited by the instrument rather than independently observed. The paper concedes this in Section 5: 'utilizing a preliminary codebook might have influenced the responses of the participants.' The circularity is partial: the 'combinations of factors' and 'evolving throughout the debugging process' themes were not directly prompted, participants added new strategies and codes, and the authors used open coding, so the central qualitative claims retain independent content.
full rationale
This is an empirical interview study rather than a derivation, so most circularity patterns (defined quantities, fitted predictions, uniqueness theorems, ansatz-via-citation) do not apply. The only load-bearing reduction is the Study 2 elicitation design: the factor-category checklist generated in Section 3.2.1 was given to participants before they described their choices, and that same checklist seeded the coding in Section 3.2.4. The authors explicitly acknowledge in Section 5 that the preliminary codebook may have influenced responses and that self-reports are subject to recall bias; these admissions bound the circularity. The paper's self-citations to prior work on mentoring and strategy effectiveness (e.g., [24, 4]) are used only to justify recruitment criteria and are not load-bearing for the central claim about contextual factor combinations. Because participants contributed new strategies (Table 4) and the 'combinations/evolution' findings emerged from narrative responses, the central claims are not fully forced by the inputs. Overall, partial circularity from instrument priming, scored 3.
Assumptions & free parameters
assumptions (4)
- domain assumption Retrospective self-reports of debugging experiences accurately reflect real-time strategy decisions.
- domain assumption The preliminary codebook built from prior work and Study 1 did not dominate the factors participants reported.
- domain assumption Years of professional experience and mentoring experience are valid proxies for debugging expertise.
- domain assumption Causal links between factors and strategies can be inferred reliably from interview segments.
invented entities (1)
-
Model of developer decision-making in web debugging strategy selection (Figure 2)
Cite this review
Pith. "Pith review of How Developers Choose Debugging Strategies for Challenging Web Application Defects." pith.science (2026). https://pith.science/paper/QW3JB4MX
@misc{pith2026250111792,
author = {Pith},
title = {Pith review of: How Developers Choose Debugging Strategies for Challenging Web Application Defects},
year = {2026},
howpublished = {\url{https://pith.science/paper/QW3JB4MX}},
note = {Machine review of arXiv:2501.11792}
}
read the original abstract
Effective debugging is a crucial aspect of software development, demanding problem-solving skills, expertise, and appropriate tools. Although previous research has studied expert developers' debugging strategies, the specific factors influencing strategy choice in complex scenarios remain underexplored. To investigate these contextual factors, we conducted two studies. First, we surveyed 35 developers to identify experiences with challenging debugging problems and contextual complexities. Second, we held semi-structured interviews with 16 experienced developers to gain deeper insight into strategic reasoning for complex debugging tasks. Insights from both groups enriched our understanding of debugging strategies at different expertise levels. We found that contextual factors interact in complex ways, and combinations of factors influence strategy choice, evolving throughout the debugging process. Hypothesis making is the baseline for debugging, with experience and code familiarity crucial for strategy selection. Our results show a gap between learning and effectively practicing strategies in challenging contexts, highlighting the need for carefully designed debugging tools and educational frameworks that align with problem contexts.
Figures
Forward citations
Cited by 1 Pith paper
-
A Grounded Theory of Debugging in Professional Software Engineering Practice
Professional debugging is an iterative process of updating a deliberately minimal mental model, using navigation, execution, and external help until a fix can be validated.
Reference graph
Works this paper leans on
-
[1]
H. Agrawal, R. A. DeMillo, and E. H. Spafford. Debugging with dynamic slicing and backtracking. Software: Practice and Experience, 23(6):589–616, 1993
work page 1993
-
[2]
M. Ahmadzadeh, D. Elliman, and C. Higgins. An analysis of patterns of debugging among novice computer science students. In Annual Conference on Innovation and Technology in Computer Science Education, 2005
work page 2005
-
[3]
A. Alaboudi and T. D. LaToza. What constitutes debugging? an exploratory study of debugging episodes. Empirical Software Engineering, 28(5):117, 2023
work page 2023
-
[4]
M. Arab, T. D. LaToza, J. Liang, and A. J. Ko. An exploratory study of sharing strategic programming knowledge. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2022. 16 HOW DEVELOPERS CHOOSE DEBUGGING STRATEGIES FOR CHALLENGING WEB APPLICATION DEFECTS A PREPRINT
work page 2022
-
[5]
M. Arab, J. Liang, Y . Yoo, A. J. Ko, and T. D. LaToza. Howtoo: A platform for sharing, finding, and using pro- gramming strategies. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 1–9. IEEE, 2021
work page 2021
- [6]
-
[7]
M. S. Carver and S. C. Risinger. Improving children’s debugging skills. In Empirical studies of programmers: Second workshop, pages 147–171, 1987
work page 1987
-
[8]
M. Decasse and A.-M. Emde. A review of automated debugging systems: Knowledge, strategies and techniques. In Proceedings.[1989] 11th International Conference on Software Engineering, pages 162–163. IEEE Computer Society, 1988
work page 1989
Show all 52 references
-
[9]
Eisenstadt
M. Eisenstadt. Tales of debugging from the front lines. In Empirical Studies of Programmers: Fifth Workshop, pages 86–112. Palo Alto, CA: Ablex Publishing Corporation, 1993
1993
-
[10]
Eisenstadt
M. Eisenstadt. My hairiest bug war stories. Communications of the ACM, 40(4):30–37, 1997
1997
-
[11]
J. Engblom. A review of reverse debugging. In Proceedings of the 2012 System, Software, SoC and Silicon Debug Conference, pages 1–6. IEEE, 2012
2012
-
[12]
J. Evans. The Pocket Guide to Debugging: Stellar Strategies for Sticky Situtations. Wizard Zines, ., 2022
2022
-
[13]
J. D. Gould. Some psychological evidence on how people debug computer programs. International Journal of Man-Machine Studies, 7(2):151–182, 1975
1975
-
[14]
J. D. Gould and P. Drongowski. An exploratory study of computer program debugging. Human Factors, 16(3):258–277, 1974
1974
-
[15]
Z. Gu, E. T. Barr, D. J. Hamilton, and Z. Su. Has the bug really been fixed? InProceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pages 55–64, 2010
2010
-
[16]
Gugerty and G
L. Gugerty and G. Olson. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 171–174, 1986
1986
-
[17]
Haidry, K
S.-e.-Z. Haidry, K. Falkner, and C. Szabo. Identifying domain-specific cognitive strategies for software engineering. In Conference on Innovation and Technology in Computer Science Education (SIGCSE), pages 206–211, 2017
2017
-
[18]
Hammer and L
D. Hammer and L. K. Berland. Confusing claims for data: A critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences, 23(1):37–46, 2014
2014
-
[19]
S. Hong, Y . Park, and M. Kim. Detecting concurrency errors in client-side java script web applications. In2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pages 61–70. IEEE, 2014
2014
-
[20]
Jiang, C
S. Jiang, C. McMillan, and R. Santelices. Do programmers do change impact analysis in debugging? Empirical Software Engineering, 22:631–669, 2017
2017
-
[21]
I. R. Katz and J. R. Anderson. Debugging: An analysis of bug-location strategies. Human-Computer Interaction, 3(4):351–399, 1987
1987
-
[22]
A. J. Ko, T. D. LaToza, S. Hull, E. A. Ko, W. Kwok, J. Quichocho, H. Akkaraju, and R. Pandit. Teaching explicit programming strategies to adolescents. In Proceedings of the 50th ACM technical symposium on computer science education, pages 469–475, 2019
2019
-
[23]
A. J. Ko and B. A. Myers. Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 151–158, 2004
2004
-
[24]
T. D. LaToza, M. Arab, D. Loksa, and A. J. Ko. Explicit programming strategies.Empirical Software Engineering, 25:2416–2449, 2020
2020
-
[25]
T. D. LaToza and B. A. Myers. On the importance of understanding the strategies that developers use. In Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering, pages 72–75, 2010
2010
-
[26]
Layman, M
L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline, and G. Venolia. Debugging revisited: Toward understanding the debugging needs of contemporary software developers. In 2013 ACM/IEEE international symposium on empirical software engineering and measurement, pages 383–392. ...
2013
-
[27]
B. Lewis. Debugging backwards in time. arXiv preprint cs/0310016, 2003. 17 HOW DEVELOPERS CHOOSE DEBUGGING STRATEGIES FOR CHALLENGING WEB APPLICATION DEFECTS A PREPRINT
2003 arXiv
-
[28]
LUCIA, F
L. LUCIA, F. Thung, D. Lo, and L. Jiang. Are faults localizable? 2012
2012
-
[29]
F. Lukey. Understanding and debugging programs.International Journal of Man-Machine Studies, 12(2):189–202, 1980
1980
-
[30]
J. W. Mickens, J. Elson, and J. Howell. Mugshot: Deterministic capture and replay for javascript applications. In NSDI, volume 10, pages 159–174, 2010
2010
-
[31]
Murphy-Hill, T
E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design space of bug fixes and how developers navigate it. IEEE Transactions on Software Engineering, 41(1):65–81, 2014
2014
-
[32]
Mutlu, S
E. Mutlu, S. Tasiran, and B. Livshits. Detecting javascript races that matter. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pages 381–392, 2015
2015
-
[33]
Nanja and C
M. Nanja and C. R. Cook. An analysis of the on-line debugging process. In Empirical studies of programmers: Second workshop, pages 172–184, 1987
1987
-
[34]
Ocariza, K
F. Ocariza, K. Bajaj, K. Pattabiraman, and A. Mesbah. An empirical study of client-side javascript bugs. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 55–64. IEEE, 2013
2013
-
[35]
F. S. Ocariza Jr, K. Pattabiraman, and B. Zorn. Javascript errors in the wild: An empirical study. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 100–109. IEEE, 2011
2011
-
[36]
S. Patel. A guide to coding qualitative data. Retrieved from, 2014
2014
-
[37]
Perscheid, B
M. Perscheid, B. Siegmund, M. Taeumel, and R. Hirschfeld. Studying the advancement in debugging practice of professional software developers. Software Quality Journal, 25:83 – 110, 2014
2014
-
[38]
Perscheid, B
M. Perscheid, B. Siegmund, M. Taeumel, and R. Hirschfeld. Studying the advancement in debugging practice of professional software developers. Software Quality Journal, 25:83–110, 2017
2017
-
[39]
M. P. Robillard, W. Coelho, and G. C. Murphy. How effective developers investigate source code: An exploratory study. IEEE Transactions on software engineering, 30(12):889–903, 2004
2004
-
[40]
Romero, B
P. Romero, B. Du Boulay, R. Cox, R. Lutz, and S. Bryant. Debugging strategies and tactics in a multi-representation software environment. International Journal of Human-Computer Studies, 65(12):992–1009, 2007
2007
-
[41]
D. I. Samudio and T. D. LaToza. Barriers in front-end web development. In 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 1–11. IEEE, 2022
2022
-
[42]
Sillito, G
J. Sillito, G. C. Murphy, and K. De V older. Questions programmers ask during software evolution tasks. In Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, pages 23–34, 2006
2006
-
[43]
Spinellis
D. Spinellis. Code reading: the open source perspective. Addison-Wesley Professional, 2003
2003
-
[44]
Spinellis
D. Spinellis. Debuggers and logging frameworks. IEEE software, 23(3):98–99, 2006
2006
-
[45]
Spinellis
D. Spinellis. Differential debugging. IEEE Software, 30(5):19–21, 2013
2013
-
[46]
Spinellis
D. Spinellis. Effective Debugging: 66 Specific Ways to Debug Software and Systems. Addison-Wesley Professional, Boston, MA, 2016
2016
-
[47]
Spinellis
D. Spinellis. Modern debugging: the art of finding a needle in a haystack. Commun. ACM, 61(11):124–134, oct 2018
2018
-
[48]
I. Vessey. Expertise in debugging computer programs: A process analysis. International Journal of Man-Machine Studies, 23(5):459–494, 1985
1985
-
[49]
M. Weiser. Program slicing. IEEE Transactions on software engineering, (4):352–357, 1984
1984
-
[50]
A. Zeller. Why programs fail: a guide to systematic debugging. Elsevier, 2009
2009
-
[51]
Zeller and R
A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. IEEE Transactions on software engineering, 28(2):183–200, 2002
2002
-
[52]
Zheng, T
Y . Zheng, T. Bao, and X. Zhang. Statically locating web application bugs caused by asynchronous calls. In Proceedings of the 20th international conference on World wide web, pages 805–814, 2011. 18
2011
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.