{"id":"d4e8442f-c1b1-49e4-a183-c10394495f3f","arxiv_id":"2505.04641","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"USRA's Feynman Quantum Academy supported 60 quantum internships from 2016 to 2024, and over 75% of its alumni remain in quantum-related roles.","lead":"This paper reviews an eight-year graduate-level quantum computing internship program run by USRA with NASA Ames, reporting where its 60 former interns now work and how much research they published. It is useful to anyone deciding whether internship programs are an effective way to build a quantum workforce.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 75% retention claim rests on an undescribed alumni-tracking procedure with an ambiguous denominator, making the paper's central quantitative claim unverifiable as presented.","rationale":"The reader identified the same load-bearing assumption: that the 60 interns and their career outcomes are measured without systematic selection bias and that the sample is complete. My concern adds a specific denominator ambiguity between '60 internships' and the number of unique students, and it emphasizes the absence of any data-collection protocol. The paper is a descriptive program review, not a hypothesis-testing study, so the lack of methodology does not invalidate its existence but does make the headline quantitative claims unverifiable. The reader's UNVERDICTED verdict is therefore appropriate; no verdict change is needed, but the authors should be asked to provide the missing measurement details if the claims are to be taken at face value.","tokens_in":26270,"tokens_out":5556,"duration_ms":67272,"concrete_test":"Request from the authors a de-identified table of all 60 internships/unique students, including current employment or education status, the source and date of each status record, and the coding rule used to classify 'quantum-related work.' Then recompute the retention rate under a conservative missing-data assumption in which all lost-to-follow-up are counted as not in quantum. If the recomputed rate falls below 75%, the headline claim is not robust to missing data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the unreported methodology behind the central claim that over 75% of students remain in quantum-related work. Section V reports this figure without defining the source cohort (60 internships vs. unique students), the coding rule for 'quantum-related work,' or the method used to ascertain current positions (e.g., LinkedIn, alumni survey, direct contact). No response rate or missing-data treatment is reported. If tracking preferentially finds alumni who remained in quantum, the observed rate is biased upward. The denominator is also ambiguous: the text reports '60 student internships' while Fig. 2 uses 60 as the number of students at the time of internship; if any students held multiple internships, the percentages are computed over an ill-defined base. Without a full roster, coding rubric, and data-collection description, the central claim cannot be checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a self-description of the USRA Feynman Quantum Academy, a graduate- and undergraduate-level internship program embedded in the NASA Ames QuAIL group, covering the period 2016–2024. After a broad review of the internship and quantum-workforce literature, the authors describe the program's structure, funding sources, student demographics, and career outcomes, and they compare the citation impact of papers with intern co-authors against papers without intern co-authors. The central quantitative claims are that the program supported 60 student internships, that over 75% of alumni remain in quantum-related work, and that intern-involved papers receive citations comparable to non-intern papers (abstract and Section V; Figures 1–3). The paper also includes a lengthy appendix summarizing the research output produced by the program.","tokens_in":26399,"tokens_out":1530,"duration_ms":21269,"significance":"If the reported outcomes are reliable, the paper would be a useful program-evaluation data point for the quantum-workforce-development literature, a field that largely relies on surveys and curricular proposals rather than longitudinal internship-outcome data. The paper's strengths are that it makes concrete, falsifiable claims (60 internships, >75% retention, citation comparability) and that it attempts to compare intern-involved publications against a non-intern baseline via citation data, an approach rarely seen in physics-education program descriptions. The literature review in Section II is broad and current. However, the evidentiary value of the central claims depends entirely on the unstated data-collection and analysis methodology, and the paper as written does not yet provide the transparency that would make those claims checkable.","major_comments":[{"comment":"The claim that 'over 75% remain in the quantum industry in some capacity' is the paper's headline result, but no methodology is given for how alumni positions were ascertained. The text does not state whether the data come from LinkedIn tracking, alumni surveys, direct contact, or a combination; it does not report a response rate or the treatment of alumni whose status could not be confirmed. If tracking preferentially reaches alumni who stayed in quantum, the 75% figure is biased upward. The authors should provide a data-collection description, a definition of the denominator (unique students vs. internships), and a coding rubric for 'quantum-related work,' including how continuing PhD students and postdocs are classified.","section":"Section V, Figure 1"},{"comment":"The denominator for the career-outcome percentages is ambiguous. The text states that the program 'supported 60 student internships' and that 'several interns have participated in multiple projects that lasted more than one year (so in some cases the totals in prior sections add up to over 60).' Yet Figure 2 labels the same quantity 60 as the number of students at the time of internship, and the percentages in Figures 1 and 2 are computed as if 60 were the number of unique students. If any student held multiple internships, the percentages in Figures 1 and 2 are over an ill-defined base, and the 75% retention claim is not reproducible. The authors should state the number of unique students explicitly and recompute all percentages over that base.","section":"Section V, paragraph 1"},{"comment":"The citation comparison between intern-participating and non-intern papers is presented as evidence that 'papers involving interns have a comparable impact... to papers without intern involvement,' but the construction of the comparison is not described. The caption refers to 'all publications (2014-April 2025) of USRA,' and the text does not specify how the two sets were matched, whether the non-intern set is intended as a control, how citation counts were normalized for publication year and venue, or how papers with both intern and non-intern authors were assigned. Without this information, the figure supports only a descriptive comparison, not the causal or comparative claim stated in the text. A specification of the inclusion criteria, matching procedure, and normalization, or a revised and more modest claim, is needed.","section":"Section III, Figure 3"},{"comment":"The opening sentence of the conclusion generalizes from the program's outcomes to a 'proven model for experiential learning in quantum computing.' Even if the retention and citation claims were fully documented, the paper compares the Feynman Academy to no counterfactual and reports no comparison group of students who did not receive this internship. The claim of a 'proven model' goes beyond what the data can support; this sentence should be softened to describe demonstrated outcomes of the program rather than a validated causal model.","section":"Section V, final paragraph"}],"minor_comments":[{"comment":"The informal title 'If You Give a Student a Quantum Internship' is engaging, but the paper would benefit from a brief program-outcome summary in the abstract that states the number of unique students, the data sources, and the period covered, so that the reader does not have to infer these from the body.","section":"Title and throughout"},{"comment":"The sentence listing employers ('Google, Amazon, IBM, JP Morgan Bank and Boeing') is presented without any counts or denominators. If this list is meant to illustrate placement outcomes, a count of alumni known to hold positions at each type of employer would make the claim more informative.","section":"Section V, paragraph 2"},{"comment":"Typographical errors occur throughout, including 'experential learning opportunities' (twice), 'serves serve as current state-of-the-art baselines' in Section III, and 'the alst decade' and 'denoisinghappen' in Appendix A. A careful proofreading pass is needed.","section":"Section III, paragraph 6"},{"comment":"The related-programs section provides useful context, but it does not compare student outcomes, selection criteria, or cost structures. Adding a short comparison table of program characteristics (duration, level, funding, mentoring structure, reported outcomes) would make the section more useful and strengthen the paper's contribution as a program review.","section":"Section IV"},{"comment":"The appendix is a long list of research summaries, but the connection between each summary and the claim that a Feynman Academy intern co-authored the work is not always stated. The reader cannot tell from the appendix alone which specific students were involved, and the mapping between the references in Appendix A and the intern/non-intern classification used in Figure 3 is not provided.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a self-description of the authors' own program, and its central quantitative claims are presented without a methodology section. This is a common issue in program-evaluation papers, and it is fixable: a detailed data-collection appendix, a clear definition of denominators, and a description of the citation-comparison procedure would address the main concerns. I do not see evidence of deliberate misreporting, but the absence of methodology makes the numbers unverifiable as they stand. The literature review is a strength and will be useful to the community once the empirical sections meet the same standard. I would recommend major revision rather than rejection because the core claims are plausible and the required information can be added within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the bottom line: this is a useful, transparent program retrospective, not a research finding. The 75% retention number is the one thing people will quote, and it is not yet supported by the reported methods. If you use the paper, use it as a case study, not as evidence.\n\nWhat it does well: the paper assembles a detailed eight-year record of the Feynman Quantum Academy — funding mix, student levels, internship structure, and a catalog of intern co-authored papers in the appendix. The literature review on quantum workforce development is current and well-cited; it does a good job placing this program alongside LANL, Fermilab, and Mitacs. The citation comparison in Fig. 3 is a fair descriptive check that intern-involved USRA papers receive citation counts comparable to non-intern USRA papers. That is not a controlled impact study, but as a program self-assessment it is a reasonable thing to show.\n\nWhere it falls short: the central claim that over 75% of interns remain in quantum-related work has no methodology. We are not told how current positions were ascertained, what response rate was achieved, how non-respondents were treated, or what coding rule defined 'quantum-related work.' The denominator is also ambiguous: Section V says 60 internships but also notes multiple internships per student, so it is unclear whether percentages are over internships or unique students. This is fixable with a short methods appendix, but as posted the headline figure cannot be checked. The citation comparison also lacks detail on how intern involvement was determined and whether any normalization was applied. The paper is self-descriptive — the authors run the program — but that alone would not worry me; the alumni outcome data and publication list are independent of the argument. The missing measurement detail is the real issue.\n\nWho it is for: people running or evaluating quantum workforce programs, and instructors looking for a concrete example of what a long-running internship can produce. It deserves a serious referee. I would send it back for revision with a required appendix on alumni tracking and citation analysis before publication.","headline":"A transparent, well-contextualized program retrospective whose headline retention figure needs a methods appendix before it can be used as evidence.","tokens_in":26925,"tokens_out":2312,"would_cite":false,"duration_ms":29529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A review of eight years of the Feynman Quantum Academy argues that graduate internships embedded in a research lab keep over 75% of students in quantum careers and produce papers cited at the same rate as the host group's other work.","keywords":["quantum workforce","internship program","experiential learning","quantum education","career outcomes","citation impact","Feynman Quantum Academy","NASA Ames"],"falsifier":"An independent career-tracking study of all 60 former interns, using contact information obtained outside the program's own records and achieving a high response rate, that found fewer than half still in quantum-related jobs would refute the 75% retention claim. Similarly, a citation comparison restricted to papers whose intern authorship is verified externally and matched to the group's non-intern papers by year and venue would test the citation-parity claim.","tokens_in":26101,"feed_emoji":"⚛️","tokens_out":4747,"duration_ms":56338,"temperature":0.7,"pith_summary":"This paper reviews eight years of the Feynman Quantum Academy, a graduate-focused internship program run inside a quantum computing research group at NASA Ames, and argues the program is a working model for preparing the quantum workforce. It reports that the program supported 60 internships between 2016 and 2024, that more than 75% of former interns still work in quantum-related roles, and that papers written with intern participation receive citations comparable to the group's papers without interns. A sympathetic reader would take the claim as: structured, research-embedded internships can both advance quantum research and launch durable careers. The paper places this program in the broader landscape of quantum workforce education and compares it with similar initiatives.","feed_headline":"Three in four quantum interns stay in quantum work","feed_subtitle":"An eight-year program review shows intern papers get citations on par with the rest of the group.","key_machinery":"The load-bearing mechanism is the internship structure itself: students join an active research group (QuAIL), work one-on-one with staff research scientists, attend group meetings, and are given access to quantum processors and high-performance computing resources. They are expected to produce code or a publication, often feeding into their doctoral thesis. The program thus functions as an apprenticeship inside a functioning research organization rather than a standalone training course.","core_discovery":"The central claim is that the Feynman Quantum Academy has demonstrated a proven model for experiential learning in quantum computing. The evidence offered is longitudinal: 60 internships funded by NASA, NSF, AFRL, DARPA, Fermilab, DLR, and DHS; a cohort that was mostly PhD students (57%) but included undergraduates, master's students, and one associate's student; a current snapshot in which more than three quarters of the alumni remain in quantum-related work; and a citation comparison in which intern-collaboration papers track the same impact as non-intern papers from the same group. The paper also documents that many alumni continued into PhDs or took positions at universities, national labs, startups, and large companies such as Google, Amazon, IBM, JP Morgan, and Boeing.","pith_inferences":["The retention figure's credibility could be tested by other programs publishing comparable longitudinal outcome data; if similar numbers appear, it would strengthen the case that structured lab internships, not selection alone, drive retention.","Because the program is embedded in a research lab with pre-existing projects, the model may be most reproducible at institutions that already run mission-driven quantum research, rather than as a standalone training add-on.","A testable extension would be comparing intern-paper citation rates against the broader field's citation distribution by year and topic, not just against the host group's other papers.","If citation parity holds, program evaluation could shift from counting placements to measuring research output, offering a quantitative template for internship assessment."],"forward_implications":["Graduate internships embedded in a research lab can feed the quantum workforce: the program reports over 75% of former interns remain in quantum-related positions.","Intern-contributed papers receive citations comparable to non-intern papers, suggesting interns produce research of similar visibility rather than peripheral work.","A single program funded by multiple agencies shows that mission-oriented funders can pool resources for workforce training.","The mix of PhD, master's, undergraduate, and associate's-level interns indicates the model can operate across educational levels.","Since many interns continue to PhDs or academic and industry positions, the program functions as a recruiting pipeline for academic and corporate quantum research."],"supporting_citations":[{"why":"Supplies longitudinal evidence that internships improve subsequent academic outcomes across disciplines.","marker":"[3]"},{"why":"Reports physics-specific benefits of hands-on research experience beyond the classroom.","marker":"[4]"},{"why":"Argues graduate students should take internships to prepare for non-academic careers.","marker":"[7]"},{"why":"Documents that most physics degree holders work outside academia, motivating off-campus training.","marker":"[8]"},{"why":"Establishes the national goal of expanding the quantum workforce pipeline that the program addresses.","marker":"[12]"},{"why":"Identifies quantum industry skills and career categories that education should target.","marker":"[13]"},{"why":"Recommends experiential training such as internships over coursework alone for quantum workforce preparation.","marker":"[16]"},{"why":"Provides the overview of QuAIL research that intern papers extend.","marker":"[2]"}],"fun_headline_variants":["Quantum internship keeps 75% of alumni in field","Intern papers match group citation impact","Most Feynman quantum interns stay in quantum","Eight-year quantum internship model shows promise","USRA quantum academy interns thrive in quantum careers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retention and citation statistics assume the program's tracking of former interns is complete and unbiased; the paper does not report how this data was collected or how nonrespondents were handled.","fun_headline_variants_meta":{"raw":{"variants":["Quantum internship keeps 75% of alumni in field","Intern papers match group citation impact","Most Feynman quantum interns stay in quantum","Eight-year quantum internship model shows promise","USRA quantum academy interns thrive in quantum careers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":1996,"prompt_tokens":800,"completion_tokens":1196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":1129}},"tokens_in":416,"tokens_out":1196,"duration_ms":11660,"temperature":1.0,"reasoning_tokens":1129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:53:46.180047+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent career-tracking study of all 60 former interns, using contact information obtained outside the program's own records and achieving a high response rate, that found fewer than half still in quantum-related jobs would refute the 75% retention claim. Similarly, a citation comparison restricted to papers whose intern authorship is verified externally and matched to the group's non-intern papers by year and venue would test the citation-parity claim.","supporting_citations":[],"review_version":1}