{"id":"1cb66185-4314-42d2-bfc4-5a1dc67ea5ac","arxiv_id":"2411.16884","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"First taxonomy of hybrid quantum-classical application issues from forum data, showing crashes and developer errors dominate.","lead":"This paper studies 531 real-world issues that developers of hybrid quantum-classical applications discussed in online forums, and organizes them into a taxonomy. It reports that most issues are crashes and that most are caused by application developers, not platforms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pooled 74% crash and 70% programmer-error rates may be artifacts of the PennyLane-dominated XDF sample; per-source breakdowns are needed before generalizing to HQC development.","rationale":"The paper's central empirical contribution is a systematically coded taxonomy and a public dataset, and that contribution is not undermined by the concern raised here. What is load-bearing is the quantitative headline: that hybrid applications are crash-dominant (74%) and that errors are predominantly introduced by application developers (70%). These numbers are pooled over two very different populations, with XDF supplying roughly 71% of the coded issues. Because XDF is a single-vendor PennyLane forum, the pooled estimates cannot be assumed to represent HQC development generally. The reader's weakest-assumption analysis already identified the PennyLane/single-screener representativeness threat; this stress-test sharpens it to a concrete, testable source-imbalance artifact. The fix is straightforward and requires only re-analysis of existing coded data: stratify Tables V and VII by forum and test homogeneity. If the stratified rates agree, the concern is resolved; if they do not, the claims must be scoped or re-weighted. Because this is an addressable empirical gap rather than a flaw in the taxonomy or the qualitative findings, the conditional verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":19380,"tokens_out":4948,"duration_ms":51888,"concrete_test":"Using the replication package, recompute the crash rate and programmer-error rate separately for XDF and QCSE, with 95% confidence intervals, and run a chi-square test of homogeneity (or Fisher's exact test) on the source-by-manifestation and source-by-cause contingency tables. If the per-source rates differ significantly or either source's confidence interval excludes the pooled estimate, the 74% and 70% claims are source artifacts and should be scoped to the PennyLane/XDF ecosystem; if the rates are homogeneous across forums, the pooled claims stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claims are computed on a pooled sample in which XDF contributes 377 of 531 coded issues (about 71%), and XDF is a vendor-specific PennyLane forum (Sec. IV.A.1, Table III). Tables V and VII report only pooled totals, so it is impossible to determine whether 74% crash-dominance and 70% programmer-caused issues are properties of HQC development as a whole or of the XDF/PennyLane ecosystem. The source skew is reinforced by the keyword list, which includes PennyLane template names such as 'AmplitudeEmbedding' (Table II). Additionally, relevance screening was performed by a single author with no reported agreement measure (Sec. IV.B.2 and Sec. VI), leaving the composition of the 1,215-thread corpus itself uncertain. The 70% programmer-error figure is also partly definitional: Developer Errors (140 issues) and most Configuration Issues (82 issues) are classified as programmer-caused by construction, so the aggregate cause distribution is not an independent estimate of root-cause responsibility. If XDF threads contain disproportionately more beginner PennyLane setup and configuration problems, the pooled percentages would overstate generalizable conclusions. The paper should condition these statistics on forum and show that the two sources are homogeneous before claiming a general characterization of HQC issues.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of 531 issues drawn from 447 discussion threads on the Xanadu Discussion Forums (XDF) and the Quantum Computing Stack Exchange (QCSE). The authors construct a four-level taxonomy of recurring issues in hybrid quantum-classical (HQC) applications, classify issues as quantum-specific, classical, or cross-domain, and analyze how issues manifest and what causes them. They report that crashes dominate (74% of studied issues) and that most issues are caused by application developers (70%). The paper closes with recommendations for application developers and platform maintainers, and it makes a replication package publicly available.","tokens_in":19618,"tokens_out":6422,"duration_ms":55552,"significance":"If the quantitative results are supported, the paper is a valuable contribution to quantum software engineering: it is among the first studies to focus on HQC application issues from developer forums, it identifies a cross-domain issue category not present in earlier quantum bug taxonomies, and it provides a labeled dataset for future research. The two-phase coding procedure, reconciliation meetings, sample-size-based coding effort, and public replication package are notable strengths. However, the headline frequencies are not currently supported as generalizable because they pool a vendor-specific PennyLane forum with a general forum, and the quantitative tables contain internal inconsistencies. The taxonomy itself and the qualitative recommendations remain useful even if the aggregate percentages require revision.","major_comments":[{"comment":"The pooled 74% crash-dominance and 70% programmer-error claims are computed on a sample in which XDF contributes 377 of 531 coded issues (about 71%), and XDF is a vendor-run PennyLane forum. Tables V and VII report only pooled totals, so it is impossible to determine whether these rates are properties of HQC development generally or of the XDF/PennyLane ecosystem. This concern is reinforced by the keyword list in Table II, which includes PennyLane template names such as 'AmplitudeEmbedding'. Please report per-source breakdowns for origin, manifestation, and cause, and test for homogeneity between XDF and QCSE (e.g., a chi-square or Fisher test) before drawing general conclusions.","section":"Sec. IV.A.1, Table III and Tables V/VII"},{"comment":"All frequencies in the paper depend on the 1,215-thread relevant corpus, but relevance screening was performed by a single author with no reported agreement measure; the threats-to-validity section acknowledges this subjectivity but does not quantify it. Please provide an inter-rater reliability check on a sample of screened threads, or at minimum discuss the direction and magnitude of screening bias that could result from the single-author procedure.","section":"Sec. IV.B.2 and Sec. VI"},{"comment":"The manifestation counts are internally inconsistent. Table V reports 483 total issues with 359 crashes, 69 incorrect outputs, 30 slow executions, 19 unknown manifestations, and 6 warnings. Table VI reports 531 total issues with 393 crashes, 79 incorrect outputs, 33 slow executions, 17 unknown manifestations, and 9 warnings. Table VIII again reports a different warning count (7) and gives a crash percentage of 2.0% for Platform Limitation, which does not match any denominator in the table. The relationship between the 483 categorized issues and the 531 total issues (including the 48 unknown-cause issues) is not explained, so the headline 74% figure cannot be verified as stated. Please reconcile these tables and specify the exact denominator for every percentage.","section":"Tables V, VI, and VIII"},{"comment":"The 70% programmer-error figure is at least partly definitional. The taxonomy defines Developer Errors (140 issues) as mistakes made by application developers and Configuration Issues (82 issues) as problems generally caused by the developer, so the dominance of Programmer Error in Table VII is built into the classification scheme. To make the causal claim meaningful, please report the cause distribution after excluding these definitional categories, or show that the cause labels were assigned independently of the issue-type taxonomy.","section":"Sec. V.D, Sec. V.A, and Table VII"}],"minor_comments":[{"comment":"The text lists 'Warnings (19)' and 'Unknown (6)', but Table V reports Warning 6 and Unknown 19; these two counts appear to be swapped and should be corrected.","section":"Sec. V.C"},{"comment":"The text says 'In an additional 48 discussion threads, we encountered issues which we could not categorize into the taxonomy', but Table III reports only 447 coded threads in total. Please clarify whether the 48 refers to threads or issues and how these relate to the 447 coded threads.","section":"Sec. V.A"},{"comment":"The percentages in Table VIII should be recomputed from the reconciled counts; several entries (e.g., the 2.0% for Platform Limitation in the Crash column) are inconsistent with both the column totals and the row totals.","section":"Table VIII"},{"comment":"The sample size calculation is mentioned only as '95% confidence interval and 5% error margin for each data source'; please provide the population sizes, the sampling method, and the resulting sample sizes to allow readers to assess representativeness.","section":"Sec. IV.C"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nI read the paper. The contribution is real: it is the first forum-based study of recurring issues in hybrid quantum-classical applications, and it introduces a category—cross-domain issues—that prior quantum-bug taxonomies from GitHub do not capture. The qualitative coding is systematic in design (two independent coders, reconciliation, pilot, sample-size calculation), and the replication package with dataset and code catalog is a concrete asset. The taxonomy itself is plausible and useful for platform teams and application developers.\n\nThe soft spots are real but mostly addressable. First, the sample is dominated by XDF: 377 of 531 issues (about 71%) come from the PennyLane-specific forum, and the keyword list includes PennyLane template names. Tables V and VII report pooled totals only, so the 74% crash rate and 70% programmer-error rate may be artifacts of that ecosystem. The paper's own external-validity note acknowledges forum choice, but the headline claims in the abstract are not conditioned on source. I would want per-source tables and a homogeneity check before calling this a general characterization of HQC development.\n\nSecond, relevance screening was done by one author with no agreement measure, and the paper explicitly prioritizes coding consistency over inter-rater agreement metrics. For a taxonomy whose counts are the main quantitative output, that is a gap, though not a fatal one. Third, there is an internal inconsistency in the manifestation counts: the text labels Warnings (19) and Unknown (6), while Table V shows Warning=6 and Unknown=19, and Table VI reports different totals (393/79/33/17/9) because it includes the 48 uncategorized issues. These need to be reconciled.\n\nOne more caveat: the 70% programmer-error figure is partly definitional, since Developer Errors and most Configuration Issues are classified as programmer-caused by construction. That is fine for a descriptive taxonomy, but it should not be read as an independent estimate of root-cause responsibility.\n\nThe stress-test concern holds up. The central taxonomy is not invalidated, but the generalizable quantitative claims are weaker than the abstract implies.\n\nWho is this for? Quantum software engineering researchers, platform maintainers, and anyone building debugging tools for hybrid applications. I would bring it to a reading group and I would cite it. I would send it to peer review—it deserves referee time—but I would ask for per-source statistics, inter-rater reliability, and a fix to the count inconsistencies before accepting.","headline":"Useful first taxonomy of hybrid quantum-classical issues from forums, but treat the headline frequencies as PennyLane-heavy sample statistics until per-source breakdowns and coding-reliability checks are added.","tokens_in":20091,"tokens_out":3287,"would_cite":true,"duration_ms":30260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hybrid quantum-classical applications fail predominantly by crashing and that most of those failures come from application-developer mistakes, not quantum hardware or platform defects.","keywords":["hybrid quantum-classical computing","software engineering","bug taxonomy","developer forums","quantum software bugs","crash analysis","empirical software engineering","cross-domain issues"],"falsifier":"Apply the same coding scheme to a fresh sample of HQC issues drawn from a different source—for example, issue trackers of non-vendor application repositories or a second forum—and compare the category proportions. If crashes do not remain the dominant manifestation (near 74%) and programmer error does not remain the leading cause (near 70%), then the paper's headline numbers are properties of its forum sample rather than of hybrid quantum-classical development.","tokens_in":19212,"feed_emoji":"🐛","tokens_out":6235,"duration_ms":52623,"temperature":0.7,"pith_summary":"The paper sets out to give the first empirically grounded account of what repeatedly goes wrong when developers build hybrid quantum-classical (HQC) applications—software that splits a computation between a quantum processor or simulator and a classical machine. After manually coding 531 real-world issues from two quantum-focused developer forums, the authors claim two headline results: HQC application failures are crash-dominant, with 74% of issues ending in a crash, and about 70% of issues are caused by mistakes made by the application developer rather than by quantum hardware or platform defects. They also claim to have identified a new class of 'cross-domain' issues that sit at the boundary between the quantum and classical subsystems and can typically be resolved by changing either side. If these claims are right, they give both application and platform developers a concrete, frequency-ordered map of where to spend effort on debugging tools, documentation, and training.","feed_headline":"Hybrid quantum apps crash in 74% of reported issues","feed_subtitle":"First taxonomy of 531 real forum failures finds developers, not hardware, cause most errors.","key_machinery":"The load-bearing object is the taxonomy itself, built from a coding pipeline: keyword search (general, quantum-specific, hybrid-algorithm, and PennyLane-template terms) over two forums, manual relevance screening of 8,278 mined threads down to 1,215 relevant ones, independent coding by two authors, iterative refinement of subcategories through open coding, and reconciliation meetings. The taxonomy carries the argument because every headline statistic—74% crashes, 70% programmer-caused errors, and the cross-domain category—is a count over its coded categories. The 'cross-domain' node is the paper's principal novel contribution: issues that manifest in either subsystem and are fixable from either side, which prior platform- and library-centered bug studies did not distinguish.","core_discovery":"The paper's central claim is that recurring problems in hybrid quantum-classical development can be organized into a four-level taxonomy of five top-level categories—Software Faults (121 issues), Library and Platform Issues (116), Developer Errors (140), Configuration Issues (82), and Hardware/Simulator Issues (24)—and that the distribution of these issues overturns a purely quantum-focused picture of quantum software bugs. The authors report that 316 of 483 classifiable issues are classical in origin, 76 are quantum-specific, and 91 are cross-domain, a category they say earlier bug taxonomies missed because those studies separated quantum and classical bugs. They find crashes are the dominant manifestation across all domains, and programmer error is the dominant cause, with platform issues second. The paper presents this taxonomy as a practical troubleshooting and testing aid, not merely a classification, and releases the coded dataset for reuse.","pith_inferences":["Because the primary forum belongs to one quantum platform vendor, the 74% and 70% figures may partly reflect that vendor's ecosystem and the kinds of questions forum users ask; re-running the coding on issues from other ecosystems or from GitHub issue trackers would test the numbers' generalizability.","If crash dominance holds across sources, HQC application development today resembles systems programming—where crashes dominate—rather than numerical software, where incorrect output typically dominates; error-handling and defensive-programming practices from systems engineering may transfer directly.","A natural next study would be to track how the cross-domain issue category maps onto API design: if most cross-domain bugs are triggered by data-shape or differentiability mismatches, platform APIs that make those contracts explicit could eliminate a measurable share of issues."],"forward_implications":["Crash prevention becomes the top priority for HQC reliability work: 359 of 483 classified issues ended in a crash, so debugging and testing tools should target crash-inducing conditions first.","Developer education and documentation may reduce failures more than hardware improvements, since 338 of 483 issues were traced to programmer error.","Cross-domain issues (91 of 483) are a distinct failure class requiring tools that understand the quantum-classical boundary, such as embedding, encoding, and hybrid algorithm design.","The taxonomy's frequency counts let testers prioritize: configuration (82), library/platform (116), and developer errors (140) are the largest actionable clusters."],"supporting_citations":[{"why":"Earlier empirical study of quantum platform bugs; supplies the platform-bug baseline and the 39.9% quantum-specific figure the paper contrasts with its hybrid findings.","marker":"[19]"},{"why":"Previous study of bug fixes in quantum programs that reported 80% of bugs as quantum-specific; the paper positions its cross-domain category as extending beyond that quantum/classical split.","marker":"[22]"},{"why":"Prior taxonomy of bugs in quantum machine learning frameworks; the closest existing work to the PennyLane-heavy data analyzed here.","marker":"[24]"},{"why":"Primary data source, the vendor-hosted discussion forum from which 293 threads were coded.","marker":"[31]"},{"why":"Secondary data source, the Stack Exchange quantum-computing board from which 154 threads were coded.","marker":"[32]"},{"why":"The hybrid quantum-classical computation library central to the primary forum; shapes the issues in the dataset.","marker":"[6]"},{"why":"The released labeled dataset, code catalog, and replication artifacts that make the taxonomy and counts auditable.","marker":"[33]"}],"fun_headline_variants":["Quantum app bugs mostly classical, taxonomy of 531 shows","74% of hybrid quantum issues end in crashes, study finds","Developer errors cause most hybrid quantum failures","First hybrid quantum failure taxonomy: classical bugs dominate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study's findings rest on the assumption that the 1,215 forum threads selected by keyword search and single-author relevance screening fairly represent the full population of hybrid quantum-classical development problems; if that sample skews toward one platform or toward beginners, the headline percentages would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Quantum app bugs mostly classical, taxonomy of 531 shows","74% of hybrid quantum issues end in crashes, study finds","Developer errors cause most hybrid quantum failures","First hybrid quantum failure taxonomy: classical bugs dominate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2391,"prompt_tokens":941,"completion_tokens":1450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1388}},"tokens_in":557,"tokens_out":1450,"duration_ms":11703,"temperature":1.0,"reasoning_tokens":1388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:45:43.261938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same coding scheme to a fresh sample of HQC issues drawn from a different source—for example, issue trackers of non-vendor application repositories or a second forum—and compare the category proportions. If crashes do not remain the dominant manifestation (near 74%) and programmer error does not remain the leading cause (near 70%), then the paper's headline numbers are properties of its forum sample rather than of hybrid quantum-classical development.","supporting_citations":[{"cited_title":"Bugs in quantum computing platforms: an empirical study,","cited_arxiv_id":null,"evidence_quote":"Earlier empirical study of quantum platform bugs; supplies the platform-bug baseline and the 39.9% quantum-specific figure the paper contrasts with its hybrid findings."},{"cited_title":"A comprehensive study of bug fixes in quantum programs,","cited_arxiv_id":null,"evidence_quote":"Previous study of bug fixes in quantum programs that reported 80% of bugs as quantum-specific; the paper positions its cross-domain category as extending beyond that quantum/classical split."},{"cited_title":"Xanadu discussion forums,","cited_arxiv_id":null,"evidence_quote":"Primary data source, the vendor-hosted discussion forum from which 293 threads were coded."},{"cited_title":"Quantum computing stack exchange forums,","cited_arxiv_id":null,"evidence_quote":"Secondary data source, the Stack Exchange quantum-computing board from which 154 threads were coded."},{"cited_title":"Online replication package,","cited_arxiv_id":null,"evidence_quote":"The released labeled dataset, code catalog, and replication artifacts that make the taxonomy and counts auditable."}],"review_version":1}