{"id":"6ac10ad7-5846-4f3b-871d-db8762f0a762","arxiv_id":"2501.11845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"PyEMILI is a new Python spectral-line identifier that improves on EMILI via a larger atomic database, refined scoring, and automatic model iteration, reaching 91.3% A-ranked agreement with manual identifications for IC 418 versus 75.3% for EMILI.","lead":"Astronomers often spend huge amounts of time matching emission lines in spectra of glowing nebulae to the elements that produce them. This paper introduces PyEMILI, a new open-source Python tool that automates this identification step and reports better results than the older EMILI program on three test objects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 75.3% to 91.3% improvement is measured against a benchmark that is neither independent nor fixed: for IC 418 the authors revised 28 original manual identifications and added 38 new ones after running PyEMILI, and the three benchmark papers share authors with this work.","rationale":"The reader's weakest assumption is that the manual identifications used as benchmarks are correct and independent, and I agree this is the soft spot. My stress-test sharpens it: the IC 418 benchmark may not even be the published manual list, because the authors revised 28 identifications and added 38 new ones after running PyEMILI, with Table 6 presenting the amendments. The Hf 2-2 benchmark is even more problematic, since the authors performed the manual identifications themselves and then compared PyEMILI to those same manual IDs; this is closer to an internal consistency check than an external validation. The paper is transparent about having no absolute benchmark, and it openly states that high agreement with manual work does not imply correctness, so the limitation is acknowledged. Still, the headline quantitative claim about improvement should be interpreted as 'better agreement with a partly author-constructed reference list' rather than 'better correctness.' The code is publicly available, which is a real strength, and the specific improvements to the atomic database and multiplet handling are concrete and testable. However, the central validation does not support a stronger verdict than conditional acceptance: an independent benchmark, ideally synthetic spectra with known ground truth, is required before the claimed improvement can be regarded as fully established. I therefore recommend no change to the reader's conditional verdict.","tokens_in":45738,"tokens_out":4353,"duration_ms":46663,"concrete_test":"Recompute Table 5 against a frozen benchmark: for IC 418 use only the original Sharpee et al. (2003) Table 3 identifications, with no substitutions from Table 6, and report the A-match rates for both EMILI and PyEMILI on exactly those 696 lines. If the 91.3% drops materially—by more than roughly the 28 re-identified plus 38 newly identified lines out of 696—the improvement is partly an artifact of benchmark revision. For an absolute correctness check, generate synthetic nebular spectra with a known input line list and physically motivated fluxes (e.g., Cloudy models with added noise at R ~ 30,000), run PyEMILI and EMILI blindly, and compare recovered assignments against the known true identifications.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PyEMILI lifts A-ranking agreement with manual identifications on IC 418 from 75.3% to 91.3%—rests on a benchmark that is not independent and is partly constructed by the code under test. In Section 3.1, after comparing with Sharpee et al. (2003), the authors state that they 're-identified 28 emission lines' and that PyEMILI 'identified 38' previously unidentified lines, with Table 6 listing these amendments. If the 696-line benchmark used for Table 5 incorporates these post-hoc changes, the 91.3% figure is not agreement with the published manual list but with a list revised using PyEMILI's own rankings. For Hf 2-2, Section 3.2.4 explicitly describes a workflow in which the same authors manually identify the lines and then compare PyEMILI against their own manual IDs (97.8% A-rate); for J0608, the benchmark is Williams et al. (2021), co-authored by a co-author of this paper. The paper itself concedes in Section 4 that 'a high level of consistency between the results of PyEMILI's identification and those of the manual identifications does not necessarily mean a high confidence level of correctness in line identifications.' Because A-ranking only means the lowest IDI among candidate IDs for a given observed line, it can be assigned even when no candidate is actually correct; the stated improvement is therefore a consistency metric, not a correctness validation. The improvement is plausible, but the presented tests do not establish it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PyEMILI is a Python rewrite and extension of the EMILI spectral-line identification code. It replaces the AtLL v2.04 atomic database with an expanded AtLL v3.00b4 database supplemented by Kurucz transition probabilities, adds a dataset of effective recombination coefficients for the H i, He i, He ii, C ii, N ii, O ii, and Ne ii nebular lines, refines an energy-bin ionization and velocity model, and provides a spectrum-analysis module for automatic line searching. The authors test the code on the PN IC 418 (comparing PyEMILI rankings against EMILI rankings, with Sharpee et al. 2003 manual IDs as a benchmark), on the PN Hf 2-2, and on the [WC11] star J0608. They report A-ranking agreement rates of 91.3% for IC 418 versus 75.3% for EMILI, 97.8% for Hf 2-2, and 91.4% for J0608.","tokens_in":46057,"tokens_out":6057,"duration_ms":60299,"significance":"If the reported improvements survive an independent validation, PyEMILI would be a valuable community tool: it is open source, runs quickly (about one minute for 1000 lines), and it assembles a curated atomic database plus effective recombination coefficients that should directly help identify faint optical recombination lines in deep spectra. The release on Zenodo/GitHub, the worked examples, and the carefully documented criteria are genuine strengths. However, the headline agreement rates are currently based on partly non-independent benchmarks, so the significance of the central claim is limited until those benchmarks are made clean.","major_comments":[{"comment":"The headline improvement from 75.3% to 91.3% for IC 418 is measured against a benchmark whose content is changed after running PyEMILI: the authors state that they re-identified 28 emission lines and that PyEMILI identified 38 previously unidentified lines, with Table 6 listing these amendments. If the 696-line comparison in Table 5 includes those 66 amendments, then about 9.5% of the benchmark has been constructed by the code under test, so the comparison with EMILI's 75.3% is not a like-for-like test. Please reproduce Table 5 using the unmodified Sharpee et al. (2003) IDs, and report the amended and newly identified lines separately.","section":"Section 3.1, Tables 5 and 6"},{"comment":"The Hf 2-2 validation compares PyEMILI with manual identifications performed by the present authors from the same spectra immediately before running PyEMILI, and the J0608 benchmark (Williams et al. 2021) has overlapping authorship with this paper. Because PyEMILI optimizes its ionization and velocity parameters from its own Sub-Line List (Section 2.2), the reported 97.8% and 91.4% agreement rates are consistency statistics against correlated references, not independent validation. The caveat in Section 4 that consistency does not imply correctness is appropriate, but it should be applied to the headline claims; an external benchmark, or a blind test with held-out lines, is needed.","section":"Sections 3.2.4 and 3.3.3, Table 8"},{"comment":"The quantity C is described as a branching ratio, but the denominator is not the sum over transitions from the upper level j; it is the sum over all transitions of the ion whose upper level is not higher than j. This is not a branching ratio and can produce very different values, because it includes transitions that are not connected to the level j. Since Eq. (8) is used to compute predicted template fluxes for the large fraction of lines without effective recombination coefficients, errors in C propagate directly into the F score and hence into A/B rankings. The comparison with literature recombination coefficients (factor of 0.2-5) is reassuring at the flux level, but the paper should either correct the denominator or demonstrate that the final rankings are insensitive to this approximation.","section":"Section 2.4.2, Eq. (9)"},{"comment":"The three extra multiplet-check rules in Section 2.4.3 (the low-flux M=3 reassignment, the W=0/F=0 M=2 reassignment, and the wedge annotation) are ad hoc components of the scoring function whose effect on the final rankings is not tested. Because the agreement rates in Tables 5 and 8 depend on how borderline lines are ranked, please provide the numbers of lines whose A/B ranking changes when these rules are disabled, or otherwise justify their inclusion with a sensitivity test.","section":"Section 2.4.3 and Table 2"},{"comment":"The abstract claims that PyEMILI gives better results than EMILI on two Galactic PNe, but for Hf 2-2 no EMILI rerun is presented; the 97.8% agreement is against the authors' own manual IDs, not against EMILI identifications. Please either provide an EMILI run on the same Hf 2-2 input line list, or restrict the improvement claim to IC 418 where the comparison with EMILI is actually made.","section":"Abstract and Section 3.2"}],"minor_comments":[{"comment":"The wording describing the 0.01 normalization factor in Eq. (2) is hard to parse; please clarify how the normalization was chosen and how N1 behaves when no Bin 1 lines are present.","section":"Section 2.2.1, Eq. (2)"},{"comment":"The text discusses an F=4 score for fluxes in the range 10^-5 to 10^-4 Imax and then states that this case is excluded, but Table 2 lists only F=0 through F=3; this explanation is confusing and should be rewritten to match the table.","section":"Section 2.4.2 and Table 2"},{"comment":"The rule that resets M=2 when W=0 and F=0 is described in prose but not encoded in Table 2; Table 2 should be updated to show the effective criteria including these overrides.","section":"Section 2.4.3, extra criterion (2)"},{"comment":"The meaning of the ellipsis in column (3) is not defined in the table notes; please state explicitly that it indicates a line that was unidentified in Sharpee et al. (2003).","section":"Table 6"},{"comment":"The statement that runtime is within one minute should be qualified with the Numba compilation caching and the machine used, since the first run may take several extra minutes.","section":"Section 2.8"},{"comment":"There is a typo in 'will sbe reported'; this should be corrected to 'will be reported'.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The central difficulty is benchmark independence. The authors have built a genuinely useful tool and have been transparent about the code and data, but the main quantitative claim (75.3% to 91.3% on IC 418) rests on a benchmark that appears to be amended with PyEMILI's own output. If the authors provide a Table 5 computed against the original Sharpee et al. (2003) identifications and add at least one external or blinded benchmark, I would be willing to support acceptance. The paper's own Section 4 caveat is honest but does not by itself fix the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: PyEMILI is a real improvement over the twenty-year-old EMILI code, with a substantially expanded atomic line list, a compiled set of effective recombination coefficients for C II, N II, O II, and Ne II, and multiplet checking that works across coupling schemes. The code and data are public on Zenodo, and the paper is honest that there is no absolute benchmark. That is the good part.\n\nWhat is actually new: the database work is the most valuable piece. Adding ~690,000 transition probabilities from Kurucz to fill gaps in Atomic Line List v3.00b4, and bundling effective recombination coefficients from the literature, will help anyone doing deep spectroscopy of PNe or H II regions. The automatic iteration using a Sub-Line List removes EMILI's requirement for a pre-identified list. Tests on three very different objects—two PNe and a [WC11] star—show plausible results, and the paper clearly states that agreement with manual IDs does not equal correctness.\n\nWhere the soft spots are, in proportion: the validation is weaker than the abstract suggests. The benchmarks are not independent: Sharpee et al. (2003) and Williams et al. (2021) include this paper's co-author, and the Hf 2-2 comparison uses the authors' own manual IDs. More importantly, for IC 418 the authors revised 28 IDs and added 38 new ones after running PyEMILI; the paper does not explicitly say whether Table 5's 91.3% counts those revisions. If it does, the improvement over EMILI is partly circular and should be stated as such. This is a fixable reporting issue, not a fatal flaw.\n\nI also have a specific technical quibble: Eq 9 for the branching ratio looks wrong. The denominator sums all transition probabilities with upper level not higher than j, which is not how a branching fraction is defined; it should sum over transitions from the same upper level j. The authors say their approximation differs from literature values by factors of 0.2–5, which is a wide band for predicted fluxes. The ad hoc scoring tweaks (25th-percentile default Aji, the extra M=2 override) could inflate A-ranking rates, though they are described openly.\n\nWho this is for: anyone who identifies lines in nebular or Wolf-Rayet spectra will find the database compilation and the code itself useful. The central claim—that PyEMILI identifies lines better than EMILI—is plausible but not rigorously established. This deserves a serious referee, provided the referee asks for clarification on the IC 418 counting, a closer look at Eq 9, and ideally one test against synthetic spectra or an independent benchmark.","headline":"PyEMILI is a genuinely useful upgrade to EMILI with a much richer atomic database, but the headline accuracy gain is measured against benchmarks that are not fully independent, so it merits refereeing with a request for cleaner validation.","tokens_in":46613,"tokens_out":2612,"would_cite":true,"duration_ms":31101,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that PyEMILI, a Python rewrite of the EMILI line identifier with a larger atomic database and effective recombination coefficients, raises agreement with manual identifications on IC 418 from 75.3% to 91.3%.","keywords":["spectral line identification","planetary nebulae","H II regions","Wolf-Rayet stars","atomic transition database","effective recombination coefficients","emission line lists","astronomy software"],"falsifier":"Take a deep echelle spectrum of a planetary nebula whose line list was published by an independent group with no overlap with the authors, run PyEMILI through the standard interface with the same 10 km/s wavelength uncertainty and default model, and compare A rankings with that independent manual list. If the agreement is well below the 91.3% reported for IC 418, the reported improvement is tied to benchmark kinship rather than to the code's general identification ability; if it stays near 90%, the method transfers.","tokens_in":45498,"feed_emoji":"🔭","tokens_out":9480,"duration_ms":85966,"temperature":0.7,"pith_summary":"PyEMILI is a reimplemented line-identification tool that aims to turn the slow, expert-driven process of identifying emission lines in deep high-dispersion spectra of planetary nebulae and H II regions into a reliable automated search. The paper claims that on the well-studied planetary nebula IC 418, the new code's top-ranked ('A') identifications agree with the published manual identifications for 91.3% of lines, against 75.3% for the older EMILI code, and that it also supplies sensible identifications for lines that were previously left unidentified. The improvements come from a much larger atomic transition database, a dedicated set of effective recombination coefficients for faint carbon, nitrogen, oxygen, and neon recombination lines, and a revised scoring system that weights wavelength, predicted flux, and multiplet coherence. If these results hold, line identification in nebular and stellar emission spectra could become faster, more reproducible, and less dependent on individual memory and experience.","feed_headline":"PyEMILI lifts line-ID success from 75% to 91% on IC 418","feed_subtitle":"A rebuilt spectral identifier with a richer atomic database agrees with manual work at 91.3 percent.","key_machinery":"The load-bearing machinery is the identification index $IDI = W + F + M$, where $W$ scores wavelength agreement, $F$ scores how well a candidate's predicted template flux matches the observed intensity, and $M$ scores how many sibling fine-structure lines from the same multiplet are present at consistent wavelengths and relative fluxes. Candidates with the lowest $IDI$ are ranked A, B, C, D, or None. Three supporting mechanisms carry the improvement. First, the energy-bin model divides the nebula into five ionization-potential bins and derives its ionization and velocity parameters from a high-confidence Sub-Line List, so no manual pre-identification is required. Second, the atomic transition database expands the Atomic Line List v3.00b4 (about 900,000 transitions) with about 690,000 transition probabilities from the Kurucz line lists, plus compiled Case B effective recombination coefficients for H I, He I, He II, C II, N II, O II, and Ne II lines. Third, the multiplet check works in LS, j-j, and intermediate coupling schemes by matching the integer term IDs assigned by the database, which lets faint heavy-element recombination lines be checked even when their coupling is not pure LS.","core_discovery":"On the paper's own terms, the central discovery is that the long-standing manual bottleneck in nebular line identification can be automated to near-expert level. PyEMILI reimplements the old EMILI scoring in Python and sharpens each of the three criteria, so that on IC 418 the fraction of lines whose manual identifications receive A ranking rises from 75.3% to 91.3%, and the share of lines with no useful ranking falls from 12.8% to 1.4%. The code re-identifies 28 IC 418 lines differently from the published table and proposes new identifications for 38 previously unidentified lines, most of them A-ranked. On Hf 2-2, 97.8% of the 411 manually identifiable lines match PyEMILI's A rankings, and on the [WC11] star J0608, 91.4% of 281 manually identified lines agree. The authors are explicit that agreement with manual lists is not itself proof of correctness; the model is anchored by strong, well-understood lines and inherits their assumptions.","pith_inferences":["The paper does not go this far, but its ranked output could serve as a clean training set for machine-learning spectral identifiers, because every line carries a grade plus a multiplet-coherence signal rather than a bare yes/no.","A testable extension is to run PyEMILI on a sample of planetary nebulae with published line lists produced by groups unrelated to the authors; if the roughly 90% agreement class holds across such samples, the method becomes a practical survey tool.","Combining the identifier with the planned MCMC recombination-line fitting would let a user move from raw spectra to electron temperature, density, and ionic abundances without manually assembling a line list, turning line identification into a plasma diagnostic pipeline."],"forward_implications":["On IC 418, the fraction of lines whose manual identifications receive A ranking rises from 75.3% (EMILI) to 91.3% (PyEMILI), and the fraction of lines with no useful ranking drops from 12.8% to 1.4%.","PyEMILI proposes new identifications for 38 previously unidentified lines in IC 418 and assigns A rankings to most of them, many supported by effective recombination coefficients or multiplet coherence.","On Hf 2-2, 97.8% of 411 manually identified lines receive A rankings matching the authors' manual work, including 100% of lines with $I_\\lambda/I_{\\mathrm{H}\\beta} \\ge 0.01$.","On the [WC11] star J0608, 91.4% of the manually identified lines agree with PyEMILI's A ranking, demonstrating applicability beyond nebulae to Wolf-Rayet stars.","The enlarged transition list and recombination-coefficient subset give the code the ability to track faint heavy-element recombination lines central to the abundance discrepancy problem in planetary nebulae."],"supporting_citations":[{"why":"Supplies the IC 418 manual line identifications used as the benchmark and introduces the original EMILI code that PyEMILI improves on.","marker":"Sharpee et al. 2003"},{"why":"Provides the Atomic Line List v3.00b4, the ~900,000-transition database that PyEMILI adopts and expands.","marker":"van Hoof 2018"},{"why":"Supplies ~690,000 transition probabilities that are matched by energy levels to complete the PyEMILI atomic database.","marker":"Kurucz 2018"},{"why":"Provides Case B effective recombination coefficients for H I and He II lines used to compute predicted template fluxes and extinction.","marker":"Storey & Hummer 1995"},{"why":"Provides He I effective recombination coefficients used in the template flux calculations.","marker":"Del Zanna & Storey 2022"},{"why":"Provides C II effective recombination coefficients, including those that identify the example line at 3918.97 Å in IC 418.","marker":"Davey et al. 2000"},{"why":"Provides N II effective recombination coefficients incorporated into the database for faint optical recombination lines.","marker":"Fang et al. 2011, 2013"},{"why":"Provides O II effective recombination coefficients for the strongest heavy-element recombination lines.","marker":"Storey et al. 2017"},{"why":"Provides Ne II effective recombination coefficients (in LS coupling) for neon recombination lines.","marker":"Kisielius et al. 1998"},{"why":"Supplies the manual line identifications of the [WC11] star J0608 against which PyEMILI is tested.","marker":"Williams et al. 2021"}],"fun_headline_variants":["PyEMILI automates nebular line ID to near-expert level","Line-ID code boosts manual match from 75% to 91%","PyEMILI: spectral line ID with 91% manual agreement","New Python identifier beats old EMILI on nebular lines","PyEMILI: near-expert line identification for nebulae"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark comparisons presuppose that the published manual identifications used for validation are correct and were made independently of PyEMILI's own scoring logic; for IC 418 and J0608 a co-author of this paper participated in producing those manual tables, so a systematic error shared by both would not be exposed.","fun_headline_variants_meta":{"raw":{"variants":["PyEMILI automates nebular line ID to near-expert level","Line-ID code boosts manual match from 75% to 91%","PyEMILI: spectral line ID with 91% manual agreement","New Python identifier beats old EMILI on nebular lines","PyEMILI: near-expert line identification for nebulae"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000434,"raw_usage":{"total_tokens":2283,"prompt_tokens":1089,"completion_tokens":1194,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":1104}},"tokens_in":705,"tokens_out":1194,"duration_ms":8158,"temperature":1.0,"reasoning_tokens":1104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:48:00.157772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a deep echelle spectrum of a planetary nebula whose line list was published by an independent group with no overlap with the authors, run PyEMILI through the standard interface with the same 10 km/s wavelength uncertainty and default model, and compare A rankings with that independent manual list. If the agreement is well below the 91.3% reported for IC 418, the reported improvement is tied to benchmark kinship rather than to the code's general identification ability; if it stays near 90%, the method transfers.","supporting_citations":[{"cited_title":"R., Storey, P","cited_arxiv_id":null,"evidence_quote":"Provides C II effective recombination coefficients, including those that identify the example line at 3918.97 Å in IC 418."},{"cited_title":"J., Sochi, T., & Bastin, R","cited_arxiv_id":null,"evidence_quote":"Provides O II effective recombination coefficients for the strongest heavy-element recombination lines."},{"cited_title":"J., Davey, A","cited_arxiv_id":null,"evidence_quote":"Provides Ne II effective recombination coefficients (in LS coupling) for neon recombination lines."},{"cited_title":"2021, ApJ, 906, 31, doi: 10.3847/1538-4357/abc754","cited_arxiv_id":null,"evidence_quote":"Supplies the manual line identifications of the [WC11] star J0608 against which PyEMILI is tested."}],"review_version":1}