{"id":"85433134-9419-4959-a27f-07cf0ca68c33","arxiv_id":"2412.00349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Simulated X-ray observations show that current relxill and relxill_nk models recover black hole parameters but leave residuals that vanish with higher numerical resolution, while full reflection spectra require proper treatment of the emission angle.","lead":"The paper tests whether two widely used X-ray reflection models will be accurate enough for data from upcoming X-ray observatories. It finds small numerical inaccuracies that can be fixed by increasing integration resolution, and a deeper limitation in how the models handle the emission angle of reflected light.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 6 chi2 values contradict the claim that NFRAD=3000 removes q=5 residuals: reduced chi2 remains 1.16-1.36 with ~10^4 dof, i.e., 12-25 sigma above expectation.","rationale":"The reader's concern about the reference ray-tracing code is legitimate and should be addressed, but the most load-bearing problem is internal: the paper's headline claim that increasing NFRAD to 3000 makes residuals disappear is contradicted by the fit statistics it reports. This is not a matter of external validation; the chi^2 values in Fig. 6 are in the text itself. If the authors can show that NFRAD=3000 is merely an intermediate value and that chi^2 converges to ~1 at larger NFRAD, the central claim survives in modified form. If not, the paper overstates the accuracy of the proposed fix. The recommendation to use NFRAD=3000 may still be a practical improvement, but the quantitative evidence as presented is insufficient. The full-reflection-spectrum result (chi^2/nu > 50 when the emission angle is not treated) is independent and appears robust, so the overall verdict remains conditional rather than reject.","tokens_in":12541,"tokens_out":9446,"duration_ms":90801,"concrete_test":"Re-fit the three q=5, a*=0.998 spectra from Fig. 6 with relline using NFRAD = 3000, 10000, and 30000 (or an adaptive quadrature integrator), holding the fit settings and responses fixed. If reduced chi^2 does not approach 1 (within about +/-0.02 for 10^4 dof) as NFRAD grows, then the residuals are not removed by increasing NFRAD and the central claim fails. Also report Delta chi^2 = chi^2(NFRAD=1000) - chi^2(NFRAD=3000) and its significance relative to the number of fitted parameters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 3.2 and Fig. 6 (right panels), the fits to the q=5 simulated spectra with NFRAD=3000 are reported as chi^2/nu = 13072.2/9966 (i=15 deg), 13527.1/9966 (i=45 deg), and 11593.0/9966 (i=75 deg). These correspond to reduced chi^2 of 1.31, 1.36, and 1.16. With nu ~ 10^4, the expected scatter is sqrt(2*nu) ~ 141, so these values are 22, 25, and 12 sigma above the mean; the p-values are astronomically small. The text states that the residuals disappear and that the models are good enough for next-generation missions, but the reported statistics show statistically unacceptable fits. The visual ratio plots (Figs. 4-5) may appear flat, yet the high count rates of X-IFU/LAD make even ~1% residuals highly significant. The paper must either demonstrate convergence of chi^2 with NFRAD (e.g., 3000 to 6000 to 12000) or revise the claim; as written, the central evidence for the proposed fix is quantitatively contradicted by the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript assesses whether the relativistic calculations in relxill and relxill_nk are accurate enough for simulated Athena/X-IFU and LAD observations. The authors use the blackray ray-tracing code to generate iron-line and full-reflection spectra for spins a*=0.5, 0.9, 0.998; inclinations i=15, 45, 75 deg; and emissivity indices q=3, 5, then fit them with relline/relline_nk or relconv*xillver/relxill in XSPEC and compare residuals and recovered parameters. They report correct parameter recovery in all cases, q=5 iron-line residuals that decrease when the number of disk interpolation points NFRAD is increased from 1000 to 3000, full-reflection residuals that require an increase of the energy-convolution parameter N_ENER_CONV to 524288, and unacceptable fits for full reflection spectra when realistic emission angles are used, leading to the recommendation that future models treat the emission angle properly.","tokens_in":12716,"tokens_out":7631,"duration_ms":68632,"significance":"If the quantitative claims held, the paper would provide a useful and reproducible benchmark for the community: it uses a public ray-tracing code, a systematic grid of simulations over spin, inclination, and emissivity index, and it identifies concrete numerical parameters (NFRAD, N_ENER_CONV) and the emission-angle treatment as the main sources of inaccuracy in current models. The explicit separation of transfer-function accuracy from atomic-table accuracy in xillver is also a helpful contribution. However, the central demonstration needs to be re-examined: as reported, the NFRAD=3000 fits for q=5 are not statistically acceptable at X-IFU/LAD count rates, so the paper's conclusion that the residual problem is solved and that the models are 'good enough' is not yet supported by its own numbers.","major_comments":[{"comment":"The claim that increasing NFRAD from 1000 to 3000 makes the q=5 residuals disappear is contradicted by the reported chi-square statistics. For a*=0.998, q=5, the right panels of Fig. 6 give chi^2/nu = 13072.2/9966 (i=15 deg), 13527.1/9966 (i=45 deg), and 11593.0/9966 (i=75 deg), i.e., reduced chi^2 of 1.31, 1.36, and 1.16. With nu ~ 10^4, the expected scatter is sqrt(2*nu) ~ 141, so these values lie about 22, 25, and 12 sigma above the mean. The same issue affects the relline_nk fits in Fig. 7. The residual plots can look flat while still indicating statistically significant model error at X-IFU/LAD count rates. Please replace the visual 'residuals disappear' criterion with a quantitative convergence test: report chi^2/nu for NFRAD = 3000, 6000, and 12000 (and, if needed, larger values) for the q=5 cases, and either demonstrate that chi^2 approaches nu or revise the conclusion to state that the models are not yet accurate enough for these data.","section":"Sec. 3.2 and Fig. 6"},{"comment":"The paper's accuracy statements are all relative to blackray, which is used as the ground truth, but no independent validation of blackray is presented in this manuscript. Section 2 specifies integration tolerances (10^-8 to 10^-6) and pixel sampling, but it does not compare computed photon trajectories, redshifts, or line profiles with analytic solutions or with a second, independent ray-tracing code. Since blackray is from the same collaboration as the models under test, a systematic error in blackray would be misattributed to relxill/relxill_nk in the residual analysis. Please add a validation test for blackray, such as a comparison of Kerr iron-line profiles with the analytic Cunningham transfer-function calculation or with an independent ray-tracing code, or explicitly identify where such validation was performed previously.","section":"Sec. 2"},{"comment":"The generality of the NFRAD fix is not established by the presented fits. The residual problem is reported in Sec. 3.1 for the q=5 grid, but the NFRAD=3000 demonstration is shown only for a*=0.998 (Figs. 4-6). Please show NFRAD=3000 fits for the remaining spins (a*=0.5 and 0.9) and report the best-fit chi^2 and recovered parameters for all 18 iron-line simulations, since the conclusion that the current models are good enough for next-generation detectors is a claim about the full parameter grid.","section":"Sec. 3.2"},{"comment":"In Case II, the conclusion that full reflection models should implement the correct emission angle rests on fitting failures with chi^2/nu > 50, but the description of the attempted improvement is incomplete. The paper states that relxill_nk uses 50 radial zones instead of relxill's one-zone approximation yet does not improve the fits, without specifying whether those 50 zones use the correct local emission angle for each zone or still average the angle within a zone. Please clarify the implementation and, if the 50-zone model is only a radial refinement, state explicitly that it does not test the emission-angle hypothesis.","section":"Sec. 3.4"}],"minor_comments":[{"comment":"The caption contains a typo: 'ray-traing code' should be 'ray-tracing code'.","section":"Fig. 4 caption"},{"comment":"The first paragraph of the Discussion contains a duplicated article: 'between the ray-tracing code calculations and the the reflection models' should read 'and the reflection models'.","section":"Sec. 4"},{"comment":"The paper does not provide a table of best-fit values and uncertainties for the 18 simulations; Fig. 1 shows only dots with error bars smaller than the symbol size, which makes it difficult to verify the claim that input parameters are recovered correctly. A supplementary table of best-fit parameters and 90% confidence ranges would improve reproducibility.","section":"Sec. 3 and Fig. 1"},{"comment":"The parameters NFRAD and N_ENER_CONV are introduced with their new recommended values, but the computational cost of changing them is not quantified anywhere in the paper. Since the Discussion recommends using the more accurate settings only when necessary, please report the runtime increase for NFRAD=3000 and N_ENER_CONV=524288.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is within scope for an astrophysical journal, and the code availability and simulation grid are strengths. I do not see a novelty-disclosure problem. The main corrective work is statistical: the reported chi-square values for the NFRAD=3000 fits contradict the paper's central claim that the residuals disappear, so the conclusion should be revised or supported by a convergence study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does something useful: it asks whether relxill and relxill_nk can handle the data quality of Athena/X-IFU and LAD without systematic bias, and it finds specific numerical fixes. Simulating iron lines and full reflection spectra with a ray-tracing code, the authors show that NFRAD=1000 produces significant residuals for steep emissivity (q=5), that raising NFRAD to 3000 removes most of them visually, and that full reflection spectra require N_ENER_CONV=524288. They also show that the angle-averaged emission treatment in current relxill/relxill_nk cannot fit spectra computed with angle-dependent emission (chi2/nu > 50). That last result is important and a bit uncomfortable for the community.\n\nThe simulation setup is careful: grids over spin, inclination, emissivity index, and the parameter recovery plots are clean. The paper is honest that it only tests relativistic calculations, not the atomic physics in xillver, and it flags the computational cost of the fixes. That is good practice.\n\nSoft spots. First, the central claim that NFRAD=3000 removes the residuals is statistically over-claimed. In Fig. 6, q=5, the reported chi2/nu values are 1.31, 1.36, and 1.16 with ~10^4 dof. Those are 12-25 sigma above expectation. The ratio plots look flat, but for X-IFU/LAD count rates even ~1% residuals are highly significant. The text says the fits are good and residuals disappear; the numbers say otherwise. The paper needs either a convergence study (NFRAD=3000, 6000, 12000) showing chi2 asymptoting to ~1, or a weaker claim that the residuals are reduced to a level of a few percent and may be acceptable for some science goals. As written, the evidence for the headline fix is quantitatively contradicted by its own statistics.\n\nSecond, the reference ray-tracing code is from the same collaboration and is not independently validated against analytic solutions or another code. The circularity burden is low because it is a forward simulation and the fits are judged by residuals, but an independent benchmark would materially strengthen the paper.\n\nThird, the scripts and updated model versions are not released. The ray-tracing code is on Zenodo, but the exact model modifications and fitting scripts are not. That makes the claimed fixes hard to verify.\n\nVerdict: this is a serious, useful paper that deserves peer review, but it needs a major revision on the statistics. The qualitative conclusion—current models need NFRAD=3000 and angle-dependent treatment for next-gen data—is plausible and likely correct. The quantitative support is not yet there. I'd send it to a journal with a request for convergence tests and release of the updated models.","headline":"Useful and honest accuracy study of relxill/relxill_nk for Athena/X-IFU and LAD, but the paper's own chi2 values undermine the claim that NFRAD=3000 fixes the residuals.","tokens_in":13370,"tokens_out":2377,"would_cite":true,"duration_ms":20813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that relxill and relxill_nk recover input black-hole parameters in simulated next-generation X-ray observations, but steep disk emissivity profiles expose numerical inaccuracies that are fixed by tripling the…","keywords":["X-ray reflection spectroscopy","relxill","relxill_nk","ray tracing","black hole spin","accretion disk emissivity","Athena/X-IFU","transfer function"],"falsifier":"Recompute the same simulated iron-line and full-reflection spectra with an independent ray-tracing code, or against analytic line shapes for a Schwarzschild black hole, and compare channel by channel; if the two reference calculations differ by as much as the residuals that $\\mathrm{NFRAD}=3000$ removes, then the reference standard itself is the problem. A second check is to fit real X-IFU-class observations of a bright Galactic black hole with steep emissivity and see whether residuals reappear when $\\mathrm{NFRAD}$ is lowered back to 1000.","tokens_in":12275,"feed_emoji":"🔭","tokens_out":8157,"duration_ms":64815,"temperature":0.7,"pith_summary":"X-ray reflection spectroscopy is the main tool for measuring black hole spin and testing General Relativity in strong gravity, and the next generation of X-ray observatories will deliver much higher-quality data than current ones. This paper asks whether the standard reflection models relxill and relxill_nk can handle that data quality. Simulating Athena/X-IFU and LAD observations of bright Galactic black holes with a precise ray-tracing code, fitting with relline and relline_nk, the paper finds the input spin, inclination, and deformation parameters are always recovered, but steep emissivity profiles ($q=5$) leave residuals that disappear only when the number of disk interpolation points $\\mathrm{NFRAD}$ is raised from 1000 to 3000. For full reflection spectra, the paper finds that the convolution energy resolution must be raised to $\\mathrm{NENER\\_CONV}=524288$, and that with the physically correct, radius-dependent emission angle neither model fits ($\\chi^2/\\nu > 50$). The conclusion is that the relativistic calculations are basically sound but need these numerical upgrades, plus a proper emission-angle treatment, for the coming high-quality data.","feed_headline":"Relxill needs triple disk grid to fit next-gen X-ray data","feed_subtitle":"Simulated Athena/X-IFU and LAD spectra show steep-emissivity residuals vanish only after tripling the disk grid.","key_machinery":"The central object is the Cunningham transfer function $f(g_*, r_e, i)$, which maps the emission integral over the observer's image plane into an integral over disk radius and the relative redshift factor $g_*$. relxill and relxill_nk precompute and tabulate this function, then integrate it numerically; the paper shows that the number of radial interpolation points in that integration, $\\mathrm{NFRAD}$, must be raised from 1000 to 3000 to capture steep emissivity profiles, and that the number of energy bins in the convolution, $\\mathrm{NENER\\_CONV}$, must be raised from 4096 to 524288 for full reflection spectra. The second mechanism is the disk emission angle $\\theta_e$: correct angle-dependent reflection requires using the angle-resolved reflection table per emission site rather than averaging the emission angle over radial zones.","core_discovery":"The paper establishes that the relativistic calculations underlying relxill and relxill_nk are accurate enough for next-generation X-ray data only after specific numerical upgrades. Fitting simulated Athena/X-IFU and LAD observations of bright Galactic black holes with relline and relline_nk always recovers the input spin and inclination, but for a steep power-law emissivity $q=5$ the fits show residuals at the iron-line low-energy tail, particularly at low inclination. Raising $\\mathrm{NFRAD}$ from 1000 to 3000 removes the residuals without changing the tabulated transfer function. For full reflection spectra, the convolution must use $\\mathrm{NENER\\_CONV}=524288$ when the emission angle equals the inclination angle; when the correct radially-varying emission angle is used, neither relxill nor relxill_nk provides an acceptable fit ($\\chi^2/\\nu > 50$), because relxill averages the emission angle over radial zones.","pith_inferences":["Beyond the paper, the residual pattern for $q=5$ at low inclination suggests the integration error is concentrated at small disk radii where the emissivity peaks; a finer or adaptive radial grid would likely fix the same problem for broken power-law emissivities, which the paper did not test.","Beyond the paper, the $\\chi^2/\\nu>50$ failure with correct emission angles implies that future reflection models will need angle-resolved xillver-style tables or ray-traced angle-dependent convolution, not just higher-resolution grids, and this may change inferred spin and ionization parameters in high-quality data.","Beyond the paper, a cheap test of the NFRAD fix is to vary the emissivity index continuously and locate the threshold $q$ where residuals appear at $\\mathrm{NFRAD}=1000$; the paper only tests $q=3$ and $q=5$.","Beyond the paper, the same accuracy checks could be applied to the lamppost coronal geometry, where the emissivity is computed rather than prescribed as a power law, to see whether the required $\\mathrm{NFRAD}$ grows worse when the illuminating flux is concentrated at the innermost stable circular orbit."],"forward_implications":["Future X-IFU and LAD iron-line fits should adopt $\\mathrm{NFRAD}=3000$ to avoid artificial residuals for steep emissivity profiles with $q>3$.","The tabulated transfer functions themselves do not need to be recomputed; the fix lies in the integration routine, so both relxill and relxill_nk can be updated by a code change rather than new FITS grids.","For full reflection spectra with $\\theta_e=i$, $\\mathrm{NENER\\_CONV}=524288$ is needed to match X-IFU and LAD data quality, otherwise residuals appear that are unrelated to the transfer function.","With physically correct, radius-dependent emission angles, current angle-averaged reflection models cannot fit next-generation data at all ($\\chi^2/\\nu>50$), so a model that uses the actual emission angle per disk zone is required.","Existing measurements with current instruments remain valid, because the angle-averaging approximation has only a minor impact on parameter estimation at present data quality."],"supporting_citations":[{"why":"Supplies the blackray ray-tracing code that generates the reference spectra and the photon-trajectory equations used as the truth model.","marker":"Abdikamalov et al. 2019"},{"why":"Provides the transfer-function formalism that relxill and relxill_nk use to rewrite the disk-integral for fast spectral calculation.","marker":"Cunningham 1975"},{"why":"Introduces relline, the relativistic line-broadening model whose accuracy is being tested.","marker":"Dauser et al. 2010"},{"why":"Defines relxill and its convolution kernel, whose NFRAD and NENER_CONV settings are under investigation.","marker":"García et al. 2014"},{"why":"Introduces relxill_nk and the non-Kerr framework used for testing General Relativity with the deformation parameter.","marker":"Bambi et al. 2017"},{"why":"Supplies the Johannsen metric with deformation parameter $\\alpha_{13}$ used in the non-Kerr fits.","marker":"Johannsen 2013"},{"why":"Describes Athena/X-IFU and provides the instrument response files used to simulate the observations.","marker":"Nandra et al. 2013"},{"why":"Describes the LAD instrument and the eXTP payload concept used for the simulated high-throughput observations.","marker":"Zhang et al. 2016"}],"fun_headline_variants":["Triple disk grid removes relxill fit residuals","Relxill needs finer grid for next-gen X-ray data","Steep emissivity exposes relxill interpolation limits","Next-gen X-ray fits demand triple relxill resolution","Grid upgrade critical for relxill accuracy in Athena era"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ray-tracing code used to generate the simulated \"true\" spectra is accurate enough to serve as the reference standard; if that code has systematic errors in photon trajectories, redshifts, or emission angles, then the residuals blamed on relxill and relxill_nk, and the proposed fixes, would be misattributed.","fun_headline_variants_meta":{"raw":{"variants":["Triple disk grid removes relxill fit residuals","Relxill needs finer grid for next-gen X-ray data","Steep emissivity exposes relxill interpolation limits","Next-gen X-ray fits demand triple relxill resolution","Grid upgrade critical for relxill accuracy in Athena era"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1421,"prompt_tokens":964,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":580,"tokens_out":457,"duration_ms":5211,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:27:51.714934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the same simulated iron-line and full-reflection spectra with an independent ray-tracing code, or against analytic line shapes for a Schwarzschild black hole, and compare channel by channel; if the two reference calculations differ by as much as the residuals that $\\mathrm{NFRAD}=3000$ removes, then the reference standard itself is the problem. A second check is to fit real X-IFU-class observations of a bright Galactic black hole with steep emissivity and see whether residuals reappear when $\\mathrm{NFRAD}$ is lowered back to 1000.","supporting_citations":[],"review_version":1}