{"id":"20c6df14-a826-4d27-9b71-6f272119407a","arxiv_id":"2505.12556","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EcoL2 combines relative error with lifecycle carbon emissions into one score, and experiments show it exposes large carbon differences among neural PDE solvers with similar accuracy.","lead":"Neural PDE solvers are compared mainly by accuracy; this paper adds the carbon cost of data generation, tuning, training, and deployment. It introduces EcoL2, a single score that balances error with emissions, and shows that equally accurate solvers can differ by orders of magnitude in carbon.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EcoL2 is not unit-invariant: with beta treated as dimensionless, recomputing Table 2 in grams instead of kilograms flips the GP/NP ranking, so the metric's 'higher is better' ordering is an artifact of the chosen mass unit.","rationale":"The reader's weakest assumption was CodeCarbon's accuracy. That is a legitimate external-tooling risk, but the larger and more fundamental problem is internal: the metric's definition is dimensionally inconsistent and unit-dependent. This does not require trusting or distrusting CodeCarbon; it is visible from Eq. (2) and Table 2 alone. Changing kg to g flips GP/NP, so the central claim that EcoL2 provides a single interpretable score with a well-defined 'higher is better' ordering fails as written. The empirical observation that similarly accurate solvers have different carbon footprints may still be true, since the advection comparison in Table 1 is large and runs on one machine, but EcoL2 as defined cannot be the vehicle for that claim until the units of beta are specified and the scores recomputed. This is a repair, not a refutation of the underlying observation, so I keep the reader's CONDITIONAL verdict rather than moving to REJECT. Agreement is partial because the reader listed dimensional inconsistency among concerns but identified CodeCarbon as the weakest assumption; I regard the unit inconsistency as more load-bearing. The paper deserves credit for raising the sustainability question and for reporting hardware and regional ablations, but the metric needs a unit-consistent formulation before its rankings are interpretable.","tokens_in":31985,"tokens_out":21684,"duration_ms":208504,"concrete_test":"Recompute every EcoL2 value in Table 2 (and, if needed, Table 1) with all carbon components converted from kg to g, leaving alpha = beta = 100 unchanged, and compare the resulting rankings with the published ones. The prediction is that GP/NP reverse: GP drops from ~0.888 to ~0.800 while NP drops from ~0.881 to ~0.853. If the ranking reverses, the metric is not well-defined unless the authors specify beta = 100 kgCO2^{-1} and rescale it to 0.1 g^{-1} under the unit change; if the ranking is stable, the concern is resolved. Report the full re-ranked tables.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (2) defines EcoL2 = (1 - e^{log_alpha R}) / (1 + beta(C_e + C_d + C_0 + C_i n_infer)). The numerator is dimensionless, while the denominator adds the pure number 1 to a sum of kgCO2 terms. For the denominator to be dimensionless, beta must carry units of kgCO2^{-1}; but the paper calls beta a hyperparameter with beta >= 1 and sets beta = 100, and its 'Dimensional analysis' paragraph asserts EcoL2 has units kgCO2^{-1}. These cannot both be right: Proposition 1 claims EcoL2 lies in (0,1), which requires the ratio to be a pure number. The consequence is concrete: because beta is treated as a dimensionless constant, the score is not invariant to the arbitrary choice of mass unit. In Table 2, using kg gives GP (C = 1.104e-6, R = 4.1e-5) EcoL2 ~0.888 and NP (C = 3.33e-7, R = 5.4e-5) ~0.881; expressing the same emissions in grams (multiplying C by 1000) while keeping beta = 100 gives GP ~0.800 and NP ~0.853, reversing the ranking. Thus the claimed 'holistic assessment' and the statement 'higher EcoL2 values are preferable' depend on a unit convention that is never specified or rescaled. This is an internal inconsistency in the metric itself, independent of CodeCarbon's measurement errors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces EcoL2, a scalar metric for neural PDE solvers that combines relative L2 error with lifecycle carbon emissions. The metric is defined in Eq. (2) as [1 - exp(log_alpha R)] / [1 + beta(Ce + Cd + C0 + Ci * n_infer)]. The authors classify emissions into embodied, developmental, operational, and inference carbon; measure them with CodeCarbon; and present experiments on PINN-family solvers and neural operators for the advection, reaction, wave, KdV, and KS equations, as well as on function approximation and symbolic regression. They prove bounds and limit statements for EcoL2 (Proposition 1 and Lemmas 1-2), show that solvers with similar relative error can have very different carbon footprints, and argue that EcoL2 should be used for sustainable model selection.","tokens_in":32318,"tokens_out":9321,"duration_ms":91881,"significance":"If the metric were unit-consistent, the paper would provide a timely and useful benchmark for Green AI in scientific machine learning. The empirical observation that solvers with comparable relative error differ substantially in CodeCarbon-measured emissions is plausible and is the paper's strongest empirical asset. The manuscript ships reproducible pseudo-code, detailed hyperparameter tables, and CodeCarbon tracking scripts, which is a definite strength. The theoretical analysis, however, is largely definitional: the bounds and limits follow immediately from the formula and do not constitute a substantive derivation. The main value of the paper lies in the proposed evaluation lens and the empirical case studies, not in the formal theory.","major_comments":[{"comment":"The definition of EcoL2 is not dimensionally consistent, and as a result the rankings it produces are not invariant to the arbitrary choice of mass unit. The numerator is dimensionless, while the denominator is 1 + beta*(... kgCO2 ...). For EcoL2 to be a pure number in (0,1), beta must carry units of kgCO2^{-1}; however, the paper treats beta as a dimensionless hyperparameter with beta >= 1 and sets beta = 100, and the 'Dimensional analysis' paragraph states that EcoL2 has units kgCO2^{-1}. These statements contradict Proposition 1. The consequence is concrete: in Table 2, recomputing the same emissions in grams while keeping beta = 100 changes GP from about 0.888 to 0.800 and NP from about 0.881 to 0.853, reversing their ranking. Because the paper's central claim is that higher EcoL2 is preferable, the metric must either specify beta's units explicitly or define C relative to a fixed reference emission so that the score is unit-invariant.","section":"Section 3, Eq. (2); 'Dimensional analysis' paragraph; Proposition 1"},{"comment":"The empirical case for EcoL2 rests on CodeCarbon's emission model C = P * t * I being accurate for all four machines and six countries, but no sensitivity or uncertainty analysis is provided. If CodeCarbon's power-draw estimates or regional carbon intensities are biased, the EcoL2 rankings and the order-of-magnitude differences reported in Tables 1 and 2 could change. The paper should report the raw power, runtime, and intensity values underlying each reported C, and should discuss or bound the sensitivity of the rankings to plausible errors in these quantities.","section":"Section 3, 'Measuring carbon'; Section 4.2"},{"comment":"The role of n_infer is ambiguous. Equation (1) defines total carbon as Ce + Cd + Co + Ci, while Eq. (2) and Algorithm 1 use Ci * n_infer. Table 1 reports values for Ci and C that are consistent with n_infer = 1, yet Figure 14 is captioned '100 Inferences'. For example, in the KdV/FNO row, Ce + Cd + Co equals the reported C to three significant digits, which is inconsistent with multiplying Ci by 100. The authors should clarify whether Ci denotes per-inference emissions or total inference emissions, and correspondingly whether Eq. (1) should read Ce + Cd + Co + Ci * n_infer.","section":"Section 3, Eqs. (1)-(2); Algorithm 1; Table 1; Figure 14"}],"minor_comments":[{"comment":"The notation for operational carbon is inconsistent: Eq. (2) uses C0, while the text, Table 1, and Table 13 use Co; the notation should be unified.","section":"Eq. (2); Section 4; Tables"},{"comment":"The sentence 'CodeCarbon estimates emissions C2' appears to contain a typo; it should read 'C' rather than 'C2'.","section":"Section 3, 'Measuring carbon'"},{"comment":"Table 1 uses the header 'MEA' while the text and Section E define 'MAE'; one spelling should be used throughout.","section":"Table 1; Section E"},{"comment":"Section 4.1 uses beta for the advection wave speed, while Section 3 uses beta as the EcoL2 weighting hyperparameter; the collision could confuse readers and one symbol should be renamed.","section":"Section 4.1; Section 3"},{"comment":"The DNN total carbon value is printed as '1.58289' without an exponent; from the preceding columns it should be 1.58289e-6.","section":"Table 13"},{"comment":"There are minor typos: 'ratioanle' in Section H.1 and 'acccuracy' in Section 1.","section":"Section H.1; Section 1"},{"comment":"Figure 3 would be easier to read if the axes were labeled and the fixed parameters for each panel were stated in the caption.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The unit-invariance problem is the main obstacle to acceptance. It is fixable by defining beta with explicit units or by normalizing carbon emissions by a reference value, and the authors should also reconcile the n_infer inconsistency between Eqs. (1)-(2), Table 1, and Figure 14. The empirical contribution is modest but appropriate for a venue interested in green and scientific machine learning; the definitional character of the theoretical section is not, by itself, a reason for rejection. I would also encourage the authors to position their metric against existing efficiency-oriented metrics in the broader ML literature, such as accuracy-per-watt or carbon-normalized accuracy measures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical message is real: similar-accuracy neural PDE solvers have very different carbon footprints, and that dimension needs to enter evaluation. I'm glad this paper exists. But the proposed EcoL2 metric as defined doesn't work as a ranking score, because it is not invariant to the units of the carbon term. The stress-test note is correct. With alpha=beta=100, Table 2's GP/NP comparison flips from GP preferred (0.888 vs 0.881) to NP preferred (0.800 vs 0.853) when you express emissions in grams rather than kilograms. The formula's numerator is dimensionless; the denominator is 1 + beta*C, so beta must carry inverse-mass units for the expression to be a number. The paper's 'Dimensional analysis' paragraph says EcoL2 has units kgCO2^{-1}, which directly contradicts Proposition 1's (0,1) claim. You can fix this by giving beta explicit units and a reference unit, or by normalizing C by a reference carbon value, but as written the 'higher is better' ordering is an artifact of the chosen mass unit.\n\nWhat the paper does well: it is the first, as far as I know, to break down carbon in neural PDE solvers into embodied, developmental, operational, and inference stages, and to measure all four across PINN and operator-learning families on five canonical PDEs. The observation that transformer-style PINNsFormer costs far more carbon than SPINN at equal accuracy is believable and well-illustrated. The hardware and country ablations are useful. The pseudo-code and supplementary materials are a good start.\n\nSofter issues: alpha and beta are hand-chosen, and the ablation only shows sensitivity, not how a user should set them. There is no comparison with existing accuracy-energy trade-off metrics. Most of the main tables lack error bars; the five-run averages only appear in the ablations. The code is 'provided with supplementary' but not yet public.\n\nThe empirical data are independent of the metric's defects, so the paper has salvageable value. But the central contribution is not correct as written. I'd send it to review, with the expectation that the authors rework the metric's units and revisit the theoretical claims. A serious referee would catch this on first read.\n\nWho is this for: researchers building benchmark suites for neural PDE solvers who want to add a sustainability axis. Worth citing the empirical measurements, not the EcoL2 score.","headline":"The paper's empirical carbon-emissions message is solid, but EcoL2 rankings flip when you change units from kg to grams, so the metric needs major revision before it can be used.","tokens_in":32834,"tokens_out":4560,"would_cite":false,"duration_ms":46173,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single metric, EcoL2, turns accuracy and lifecycle carbon emissions into one score, and reorders neural PDE solvers once emissions are counted.","keywords":["EcoL2 metric","neural PDE solvers","carbon footprint","physics-informed neural networks","neural operators","lifecycle carbon assessment","sustainable machine learning","relative L2 error"],"falsifier":"Re-run the six benchmark solvers on the same four machines while recording energy draw with an external power meter, and compute EcoL2 using measured energy and independent regional grid intensities; if the ordering of PINNsFormer versus SPINN on advection or CNO versus DON on Kuramoto-Sivashinsky changes, or if the tracker's kgCO2 values deviate from the meter beyond noise, the paper's central empirical demonstration does not survive.","tokens_in":31770,"feed_emoji":"♻️","tokens_out":6623,"duration_ms":63890,"temperature":0.7,"pith_summary":"The paper argues that evaluating neural PDE solvers by accuracy alone is misleading, because solvers with nearly identical relative error can differ by large factors in carbon emissions. It introduces EcoL2, a score that combines relative L2 error with lifecycle carbon, summing embodied, developmental, operational, and inference emissions, with higher values meaning better combined performance. On canonical PDE benchmarks the score reorders the field: for the advection equation, PINNsFormer and SPINN have comparable relative error but EcoL2 scores of 0.022 versus 0.103, driven mainly by PINNsFormer's higher tuning and training emissions. The paper also shows the metric responds to application priorities through two weighting hyperparameters, and that the same solver's score changes with hardware and with the carbon intensity of the electricity grid where it runs. If the empirical demonstration holds, EcoL2 gives the community a way to compare solvers on both predictive performance and long-term environmental cost.","feed_headline":"One score ranks neural PDE solvers by error and carbon","feed_subtitle":"EcoL2 merges relative L2 error with lifecycle emissions; accuracy alone hides large carbon gaps between similar solvers.","key_machinery":"The load-bearing object is the EcoL2 ratio itself. Its numerator, $1-e^{\\log_\\alpha R}=1-R^{1/\\ln\\alpha}$, maps relative error into $(0,1)$ with diminishing returns as error shrinks; its denominator, $1+\\beta(C_e+C_d+C_o+C_i\\,n_{\\text{infer}})$, scales the full lifecycle carbon bill by $\\beta$. The carbon sum is what makes the metric new: it forces data generation, hyperparameter search, final training, and repeated deployment to be counted alongside error. Proofs that the score is bounded and moves in the intended directions, together with the tunable hyperparameters, are what let the metric serve both high-accuracy and low-carbon regimes.","core_discovery":"The central discovery is that accuracy and carbon footprint are separable dimensions of a neural PDE solver's worth, and that a single scalar can carry both. EcoL2 is defined as $$\\text{EcoL2} = \\frac{1 - $e^{{\\log_\\alpha R}}$}{1 + \\$\\beta$(C_e + C_d + C_o + C_i \\cdot n_{\\text{infer}})},$$ where $R$ is relative L2 error, the $C$ terms are embodied, developmental, operational, and inference carbon, and $\\alpha,\\beta$ weight accuracy versus sustainability. The paper proves the score lies in $(0,1)$, approaches 1 exactly when $R\\to 0$ and total carbon $C\\to 0$, and approaches 0 when the solver is inaccurate or the carbon bill grows without bound. Across advection, reaction, wave, KdV, and Kuramoto-Sivashinsky equations, the empirical claim is that accuracy-only rankings hide large emission differences, so comparable models can receive very different EcoL2 scores. The paper treats EcoL2 as a general evaluation protocol rather than a PDE-specific benchmark, applying the same score to function approximation and symbolic regression.","pith_inferences":["The denominator's carbon terms are dimensionful, so EcoL2's numerical value depends on the mass unit chosen for CO2; if the field adopts the metric, a standard reporting unit such as kgCO2 will be necessary for scores to be comparable across papers.","Because the proof of Lemma 1 requires $R\\to 0$ and $C\\to 0$ simultaneously, EcoL2 will never reach 1 in practice, so useful thresholds would need to be calibrated on existing benchmarks.","The strong dependence on developmental carbon suggests that cheaper tuning strategies, such as shared hyperparameter sweeps or prior-informed search, could improve EcoL2 as much as better architectures; that hypothesis is testable but not established by the paper.","Once EcoL2 becomes a standard reporting target, teams that report only error will be visibly hiding carbon, a practical reversal of the paper's own argument that a metric which becomes the target can stop being a good measure."],"forward_implications":["EcoL2 turns model selection into a two-criterion decision: a solver with slightly worse error can be the better choice if its lifecycle carbon is far lower, and the paper gives specific cases where this flips the ranking.","Hyperparameter tuning emerges as a dominant emission term for PINN-family solvers, so reporting EcoL2 pushes developers to count tuning cost rather than only final training error.","Because $\\alpha$ and $\\beta$ are tunable, the same benchmark can rank the best solver differently for accuracy-first applications versus large-scale deployment, making the metric explicitly application-dependent.","Hardware and geography enter the score through power draw and grid carbon intensity, so EcoL2 provides a way to compare not only algorithms but also the conditions under which they run.","The metric carries over to non-PDE scientific machine-learning tasks such as function approximation and symbolic regression, implying it could serve as a general accuracy-carbon score for scientific models."],"supporting_citations":[{"why":"Supplies the emission estimate $C=P\\times t\\times I$ used to measure every carbon component in the EcoL2 experiments.","marker":"[46]"},{"why":"Defines physics-informed neural networks, the baseline family whose relative error and emissions are scored on advection, reaction, and wave equations.","marker":"[15]"},{"why":"Provides PINNsFormer, the transformer-based baseline whose high developmental carbon drives the low EcoL2 score.","marker":"[16]"},{"why":"Provides SPINN, the low-carbon baseline with nearly identical accuracy to PINNsFormer on advection.","marker":"[17]"},{"why":"Provides the deep operator network (DON) baseline scored on the KdV and KS equations.","marker":"[18]"},{"why":"Provides the Fourier neural operator (FNO) baseline used for the main tables and the country and hardware ablations.","marker":"[19]"},{"why":"Provides the convolutional neural operator (CNO) baseline, the accuracy-comparable alternative to DON on KS.","marker":"[20]"},{"why":"Supplies the pseudospectral data-generation code that produces operator training data and thus the embodied-carbon component.","marker":"[67]"},{"why":"Supplies the embodied-versus-operational carbon distinction from structural engineering that the lifecycle accounting extends to AI.","marker":"[23]"}],"fun_headline_variants":["EcoL2: A single score for neural PDE accuracy and carbon","Beyond error: EcoL2 brings carbon into neural PDE rankings","Judge neural PDE solvers on emissions as well as error","New metric EcoL2 scores neural PDE solvers by both error and carbon","Carbon-aware scoring for neural PDE solvers: EcoL2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every EcoL2 value inherits the accuracy of the emission estimate $C = P \\times t \\times I$ that the paper's tracker derives from power draw, runtime, and regional grid carbon intensity; if that estimate is unreliable for a solver or machine, the empirical rankings EcoL2 produces inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["EcoL2: A single score for neural PDE accuracy and carbon","Beyond error: EcoL2 brings carbon into neural PDE rankings","Judge neural PDE solvers on emissions as well as error","New metric EcoL2 scores neural PDE solvers by both error and carbon","Carbon-aware scoring for neural PDE solvers: EcoL2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1922,"prompt_tokens":944,"completion_tokens":978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":888}},"tokens_in":560,"tokens_out":978,"duration_ms":8739,"temperature":1.0,"reasoning_tokens":888,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:31:47.819744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six benchmark solvers on the same four machines while recording energy draw with an external power meter, and compute EcoL2 using measured energy and independent regional grid intensities; if the ordering of PINNsFormer versus SPINN on advection or CNO versus DON on Kuramoto-Sivashinsky changes, or if the tracker's kgCO2 values deviate from the meter beyond noise, the paper's central empirical demonstration does not survive.","supporting_citations":[{"cited_title":"Codecarbon: estimate and track carbon emissions from machine learning computing.Cited on, 20, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the emission estimate $C=P\\times t\\times I$ used to measure every carbon component in the EcoL2 experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines physics-informed neural networks, the baseline family whose relative error and emissions are scored on advection, reaction, and wave equations."},{"cited_title":"Pinnsformer: A transformer-based framework for physics-informed neural networks","cited_arxiv_id":null,"evidence_quote":"Provides PINNsFormer, the transformer-based baseline whose high developmental carbon drives the low EcoL2 score."},{"cited_title":"Convolutional neural operators for robust and accurate learning of PDEs.Advances in Neural Information Processing Systems, 36: 77187–77200, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the convolutional neural operator (CNO) baseline, the accuracy-comparable alternative to DON on KS."},{"cited_title":"Automating embodied and operational carbon assessment in urban sustainable development.Automation in Construction, 176:106245, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the embodied-versus-operational carbon distinction from structural engineering that the lifecycle accounting extends to AI."}],"review_version":1}