Pith. sign in

REVIEW 2 major objections 48 references

Choosing What to Calibrate and What to Estimate in Structural Models

T0 review · 2 major / 0 minor · reviewed 2026-06-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sensitivity statistic selects the calibration-estimation split in structural models that minimizes worst-case local bias in targets such as policy effects.

desk verdict The paper gives a clean derivative-based procedure for picking the calibration-estimation split that keeps a target like a policy effect least sensitive to calibration mistakes. read the letter →

arxiv 2606.25688 v1 pith:Z3GKWNRH submitted 2026-06-24 econ.EM

Joan Alegre Canton This is my paper · ORCID
classification econ.EM
keywords calibrationestimationpartitionsensitivitystatisticstructuralmodelslocalbiasNewKeynesianmodelparameterselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structural models conventionally fix some parameters by calibration and estimate the rest, but the paper treats the split itself as a selection problem. For each admissible partition it builds a scalar sensitivity statistic that records the local response of a chosen target object to small changes in the calibrated parameters. The procedure then keeps the partition whose statistic is smallest, which directly bounds the largest local bias that calibration errors can produce. Only first derivatives are required, so the method avoids repeated full re-estimation and applies to any structural model whose target object can be differentiated with respect to parameters. An application to the Nakamura-Steinsson New Keynesian model shows that the choice of partition can turn a reliable result into one that is highly sensitive to small calibration mistakes.

What carries the argument

the sensitivity statistic, a scalar that records the local response of the target object to perturbations of the calibrated parameters

What would settle it

In a Monte Carlo exercise where true parameter values are known, compute the actual bias in the target object under calibrated values drawn from a plausible error distribution and verify whether the partition chosen by the statistic produces smaller bias than the next-best partitions.

Watch

Extended reading notes

Core claim

For any structural model and target object the paper defines, for every admissible calibration-estimation partition, a scalar sensitivity statistic equal to the local derivative response of the target to perturbations in the calibrated parameters; the partition that produces the smallest value of this statistic is selected because it minimizes the worst-case local bias that can arise from errors in the calibrated values.

Load-bearing premise

The local linear approximation given by the derivatives of the target with respect to calibrated parameters accurately captures the size of bias that calibration errors induce.

Editorial extensions

If this is right

  • The procedure applies to any target object whose derivatives with respect to parameters exist, including policy effects, welfare measures, impulse responses, and treatment effects.
  • In the New Keynesian application some partitions remain reliable under large miscalibrations while others generate large bias from small errors.
  • The method requires only local derivatives and therefore scales to models where repeated re-estimation would be costly.
  • Partition choice is shown to have first-order consequences for the credibility of model-based conclusions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same statistic could be used to rank which parameters are most worth estimating when data are scarce.
  • If the local linear approximation proves inadequate, the framework could be extended to higher-order or global sensitivity measures.
  • The approach supplies a quantitative criterion that could be added to existing robustness checks in applied structural work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper treats the choice of which parameters to calibrate versus estimate in structural models as a partition-selection problem. For each admissible partition it constructs a scalar sensitivity statistic from the local derivatives of a target object (policy effect, welfare, impulse response, etc.) with respect to the calibrated parameters. The partition that minimizes this statistic is selected on the grounds that it minimizes worst-case local bias from calibration errors. The procedure is illustrated in two canonical examples and then applied to the New Keynesian model of Nakamura and Steinsson (2018), where the authors report that some partitions remain reliable under sizeable miscalibrations while others generate large bias from small calibration errors. The method uses only local derivatives and avoids repeated re-estimation.

Significance. If the local linear approximation is shown to be reliable at the error magnitudes the authors deem relevant, the procedure supplies a computationally light, systematic criterion for partition choice that could improve the credibility of structural-model results, especially in applications where the target object is a policy or welfare quantity. The explicit demonstration that partition choice can materially affect robustness in the NK setting is a useful illustration. The fact that the statistic is obtained from local derivatives without re-estimation is a practical strength.

major comments (2)
  1. [Abstract; Nakamura-Steinsson (2018) application] Abstract and Nakamura-Steinsson application: the central claim is that the partition minimizing the sensitivity statistic also minimizes worst-case local bias from calibration errors. The statistic is constructed from first-order derivatives, so the bound on bias is exact only in the infinitesimal limit. The manuscript provides no analytic derivation or numerical check confirming that the linear ranking survives at the finite miscalibration sizes illustrated in the NK example.
  2. [Abstract; Nakamura-Steinsson (2018) application] The weakest assumption is that the local linear approximation via derivatives of the target with respect to calibrated parameters accurately reflects the impact of calibration errors. Higher-order terms or curvature could reverse the ranking for the "sizeable" miscalibrations the paper considers; no verification of this approximation is reported.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. The two major comments raise a valid point about the scope of the local approximation. We address them point by point below and commit to revisions that directly respond to the concern.

read point-by-point responses

  1. Referee: [Abstract; Nakamura-Steinsson (2018) application] Abstract and Nakamura-Steinsson application: the central claim is that the partition minimizing the sensitivity statistic also minimizes worst-case local bias from calibration errors. The statistic is constructed from first-order derivatives, so the bound on bias is exact only in the infinitesimal limit. The manuscript provides no analytic derivation or numerical check confirming that the linear ranking survives at the finite miscalibration sizes illustrated in the NK example.

    Authors: We agree that the first-order sensitivity statistic delivers an exact bound on bias only in the infinitesimal limit; the paper states this explicitly by referring to 'local bias' and 'local derivatives.' The NK application is intended to illustrate that partitions can differ dramatically in their local sensitivity, not to claim that the local ranking is necessarily preserved for every finite perturbation. In the revision we will add a new subsection that numerically evaluates the actual (non-local) bias for the finite miscalibration magnitudes shown in the application and reports whether the ordering of partitions induced by the sensitivity statistic is preserved. revision: yes

  2. Referee: [Abstract; Nakamura-Steinsson (2018) application] The weakest assumption is that the local linear approximation via derivatives of the target with respect to calibrated parameters accurately reflects the impact of calibration errors. Higher-order terms or curvature could reverse the ranking for the "sizeable" miscalibrations the paper considers; no verification of this approximation is reported.

    Authors: The referee is correct that higher-order terms or curvature could, in principle, reverse the local ranking for the finite miscalibrations examined. Because the procedure is deliberately local and derivative-based, it cannot automatically guarantee global robustness. We will therefore include, in the revised manuscript, a direct numerical check within the Nakamura-Steinsson example that compares the realized bias under the reported finite perturbations across the candidate partitions and discusses any discrepancies with the local ranking. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; procedure is a direct definition of sensitivity-based selection

full rationale

The paper defines the sensitivity statistic explicitly from local derivatives of the target object w.r.t. calibrated parameters and selects the minimizing partition by that definition. This is a methodological proposal whose central claim follows immediately from the construction of the statistic as a local-bias measure; it does not reduce any independent prediction or first-principles result to its own inputs. No self-citation chains, fitted inputs renamed as predictions, or ansatzes smuggled via prior work appear in the load-bearing steps. The derivation is therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The approach rests on the domain assumption that local derivatives suffice to measure sensitivity and that the target object is differentiable with respect to calibrated parameters.

assumptions (1)
  • domain assumption Local derivatives of the target object with respect to calibrated parameters capture the relevant response to calibration errors
    The sensitivity statistic is constructed from these derivatives as described in the abstract.
invented entities (1)
  • sensitivity statistic
    purpose: Scalar measure of local response of a target object to perturbations in calibrated parameters
    Newly defined quantity used to rank partitions; no independent evidence provided in abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Choosing What to Calibrate and What to Estimate in Structural Models." pith.science (2026). https://pith.science/paper/Z3GKWNRH

@misc{pith2026260625688,
  author       = {Pith},
  title        = {Pith review of: Choosing What to Calibrate and What to Estimate in Structural Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3GKWNRH}},
  note         = {Machine review of arXiv:2606.25688}
}
read the original abstract

Structural models often fix (calibrate) some parameters and estimate the rest, but this calibration-estimation partition is usually chosen by convention. This paper treats that choice as an econometric partition-selection problem. For each admissible partition, we construct a scalar sensitivity statistic measuring the local response of a target object -- such as a policy effect, welfare measure, impulse response, or treatment effect -- to perturbations of the calibrated parameters. The selected partition minimizes this statistic and therefore minimizes worst-case local bias from calibration errors. We first illustrate the decision problem in two canonical examples. We then apply it to the New Keynesian model of Nakamura and Steinsson (2018), where the partition choice has large implications for credibility: some partitions remain reliable under sizeable miscalibrations, whereas others generate large bias from small calibration errors. The procedure requires only local derivatives, avoids repeated re-estimation, and applies to a broad class of structural models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 3 canonical work pages

  1. [1]

    Journal of Business & Economic Statistics, 28 (2), 201--218

    Aguirregabiria, Victor (2010), Another look at the identification of dynamic discrete decision processes: An application to retirement behavior. Journal of Business & Economic Statistics, 28 (2), 201--218. aguirregabiria2010another

    2010

  2. [2]

    Journal of Econometrics, 168 (1), 156--173

    Aguirregabiria, Victor and Chun-Yu Ho (2012), A dynamic oligopoly game of the us airline industry: Estimation and policy experiments. Journal of Econometrics, 168 (1), 156--173. aguirregabiria2012dynamic

    2012

  3. [3]

    Quantitative Marketing and Economics, 12, 267--304

    Aguirregabiria, Victor and Junichi Suzuki (2014), Identification and counterfactuals in dynamic models of market entry and exit. Quantitative Marketing and Economics, 12, 267--304. aguirregabiria2014identification

    2014

  4. [4]

    arXiv preprint arXiv:2310.05761

    Alegre, Joan and Juan Carlos Escanciano (2023), Robust minimum distance inference in structural models. arXiv preprint arXiv:2310.05761. alegre2023robust

    work page arXiv 2023

  5. [5]

    Econometric reviews, 26 (2-4), 113--172

    An, Sungbae and Frank Schorfheide (2007), Bayesian analysis of dsge models. Econometric reviews, 26 (2-4), 113--172. an2007bayesian

    2007

  6. [6]

    The Quarterly Journal of Economics, 132 (4), 1553--1592

    Andrews, Isaiah, Matthew Gentzkow, and Jesse M Shapiro (2017), Measuring the sensitivity of parameter estimates to estimation moments. The Quarterly Journal of Economics, 132 (4), 1553--1592. andrews2017measuring

    2017

  7. [7]

    Econometrics and Statistics, 30, 36--59

    Antoine, Bertille and Eric Renault (2024), Gmm with nearly-weak identification. Econometrics and Statistics, 30, 36--59. antoine2024gmm

    2024

  8. [8]

    Journal of Econometrics, 215 (2), 473--485

    Arcidiacono, Peter and Robert A Miller (2020), Identifying dynamic discrete choice models off short panels. Journal of Econometrics, 215 (2), 473--485. arcidiacono2020identifying

    2020

Show all 48 references
  1. [9]

    Quantitative Economics, 12 (1), 77--108

    Armstrong, Timothy B and Michal Koles \'a r (2021), Sensitivity analysis using approximate moment condition models. Quantitative Economics, 12 (1), 77--108. armstrong2021sensitivity

    2021

  2. [10]

    The RAND Journal of Economics, 46 (1), 103--145

    Barwick, Panle Jia and Parag A Pathak (2015), The costs of free entry: an empirical study of real estate agents in greater boston. The RAND Journal of Economics, 46 (1), 103--145. barwick2015costs

    2015

  3. [11]

    Available at SSRN 4784452

    Beresteanu, Arie, Paul B Ellickson, Sanjog Misra, and James C Reeder III (2024), The dynamics of retail oligopoly. Available at SSRN 4784452. beresteanu2024dynamics

    2024

  4. [12]

    Quantitative Economics, 13 (3), 907--954

    Bonhomme, St \'e phane and Martin Weidner (2022), Minimizing sensitivity to model misspecification. Quantitative Economics, 13 (3), 907--954. bonhomme2022minimizing

    2022

  5. [13]

    arXiv preprint arXiv:2109.08109

    Cocci, Matthew D and Mikkel Plagborg-M ller (2021), Standard errors for calibrated parameters. arXiv preprint arXiv:2109.08109. cocci2021standard

    2021

  6. [14]

    US Census Bureau Center for Economic Studies Paper No

    Collard-Wexler, Allan (2011), Productivity dispersion and plant selection in the ready-mix concrete industry. US Census Bureau Center for Economic Studies Paper No. CES-WP-11-25. collard2011productivity

    2011

  7. [15]

    Econometrica, 81 (3), 1003--1037

    Collard-Wexler, Allan (2013), Demand fluctuations in the ready-mix concrete industry. Econometrica, 81 (3), 1003--1037. collard2013demand

    2013

  8. [16]

    Review of Economics and Statistics, 94 (1), 260--272

    Conley, Timothy G, Christian B Hansen, and Peter E Rossi (2012), Plausibly exogenous. Review of Economics and Statistics, 94 (1), 260--272. conley2012plausibly

    2012

  9. [17]

    Econometrica, 75 (3), 837--873

    Das, Sanghamitra, Mark J Roberts, and James R Tybout (2007), Market entry costs, producer heterogeneity, and export dynamics. Econometrica, 75 (3), 837--873. das2007market

    2007

  10. [18]

    Technical report, cemmap working paper

    Escanciano, Juan Carlos and Lin Zhu (2013), Set inferences and sensitivity analysis in semiparametric conditionally identified models. Technical report, cemmap working paper. escanciano2013set

    2013

  11. [19]

    Journal of Econometrics, 238 (1), 105552

    Forneron, Jean-Jacques (2024), Detecting identification failure in moment condition models. Journal of Econometrics, 238 (1), 105552. forneron2024detecting

    2024

  12. [20]

    The Quarterly Journal of Economics, 137 (3), 1299--1344

    Hazell, Jonathon, Juan Herreno, Emi Nakamura, and J \'o n Steinsson (2022), The slope of the phillips curve: evidence from us states. The Quarterly Journal of Economics, 137 (3), 1299--1344. hazell2022slope

    2022

  13. [21]

    Journal of Applied Econometrics, 35 (7), 797--813

    Honore, Bo, Thomas J rgensen, and Aureo de Paula (2020), The informativeness of estimation moments. Journal of Applied Econometrics, 35 (7), 797--813. honore2020informativeness

    2020

  14. [22]

    Journal of Political Economy, 125 (3), 798--847

    Igami, Mitsuru (2017), Estimating the innovator’s dilemma: Structural analysis of creative destruction in the hard disk drive industry, 1981--1998. Journal of Political Economy, 125 (3), 798--847. igami2017estimating

    2017

  15. [23]

    Journal of Monetary Economics, 57 (2), 189--202

    Iskrev, Nikolay (2010), Local identification in dsge models. Journal of Monetary Economics, 57 (2), 189--202. iskrev2010local

    2010

  16. [24]

    Journal of Economic Dynamics and Control, 99, 54--81

    Iskrev, Nikolay (2019), What to expect when you're calibrating: Measuring the effect of calibration on the estimation of macroeconomic models. Journal of Economic Dynamics and Control, 99, 54--81. iskrev2019expect

    2019

  17. [25]

    Review of Economics and Statistics, 105 (2), 474--481

    J rgensen, Thomas H (2023), Sensitivity to calibrated parameters. Review of Economics and Statistics, 105 (2), 474--481. jorgensen2023sensitivity

    2023

  18. [26]

    Review of Economic Studies, rdag039

    Kalouptsidi, Myrto, Yuichi Kitamura, Lucas Lima, and Eduardo Souza-Rodrigues (2026), Counterfactual analysis for structural dynamic discrete choice models. Review of Economic Studies, rdag039. kalouptsidi2026counterfactual

    2026

  19. [27]

    Quantitative Economics, 12 (2), 351--403

    Kalouptsidi, Myrto, Paul T Scott, and Eduardo Souza-Rodrigues (2021), Identification of counterfactuals in dynamic discrete choice models. Quantitative Economics, 12 (2), 351--403. kalouptsidi2021identification

    2021

  20. [28]

    Econometrica, 81 (3), 1185--1201

    Kitamura, Yuichi, Taisuke Otsu, and Kirill Evdokimov (2013), Robustness, infinitesimal neighborhoods, and moment restrictions. Econometrica, 81 (3), 1185--1201. kitamura2013robustness

    2013

  21. [29]

    Journal of Econometrics, 236 (2), 105477

    Koci e cki, Andrzej and Marcin Kolasa (2023), A solution to the global identification problem in dsge models. Journal of Econometrics, 236 (2), 105477. kocikecki2023solution

    2023

  22. [30]

    Econometrica, 79 (6), 1995--2032

    Komunjer, Ivana and Serena Ng (2011), Dynamic identification of dynamic stochastic general equilibrium models. Econometrica, 79 (6), 1995--2032. komunjer2011dynamic

    2011

  23. [31]

    manuscript

    Kryukov, Yaroslav (2010), Dynamic r&d and the effectiveness of policy intervention in the pharmaceutical industry. manuscript. Tepper School of Business. Carnegie Mellon University. kryukov2010dynamic

    2010

  24. [32]

    arXiv preprint arXiv:2408.16330

    Lau, Chun Pong (2024), Sensitivity analysis for dynamic discrete choice models. arXiv preprint arXiv:2408.16330. lau2024sensitivity

    2024

  25. [33]

    International Economic Review, 56 (4), 1261--1290

    Lin, Haizhen (2015), Quality choice and market structure: A dynamic analysis of nursing home oligopolies. International Economic Review, 56 (4), 1261--1290. lin2015quality

    2015

  26. [34]

    NBER macroeconomics annual, 20, 313--366

    Lubik, Thomas and Frank Schorfheide (2005), A bayesian look at new open economy macroeconomics. NBER macroeconomics annual, 20, 313--366. lubik2005bayesian

    2005

  27. [35]

    The Quarterly Journal of Economics, 133 (3), 1283--1330

    Nakamura, Emi and J \'o n Steinsson (2018), High-frequency identification of monetary non-neutrality: the information effect. The Quarterly Journal of Economics, 133 (3), 1283--1330. nakamura2018high

    2018

  28. [36]

    Journal of econometrics, 29 (3), 229--256

    Newey, Whitney K (1985), Generalized method of moments specification testing. Journal of econometrics, 29 (3), 229--256. newey1985generalized

    1985

  29. [37]

    Handbook of econometrics, 4, 2111--2245

    Newey, Whitney K and Daniel McFadden (1994), Large sample estimation and hypothesis testing. Handbook of econometrics, 4, 2111--2245. newey1994large

    1994

  30. [38]

    Review of Economic Studies, 81 (3), 1229--1262

    Norets, Andriy and Xun Tang (2014), Semiparametric inference in dynamic binary choice models. Review of Economic Studies, 81 (3), 1229--1262. norets2014semiparametric

    2014

  31. [39]

    Quantitative Economics, 3 (1), 95--132

    Qu, Zhongjun and Denis Tkachenko (2012), Identification and frequency domain quasi-maximum likelihood estimation of linearized dynamic stochastic general equilibrium models. Quantitative Economics, 3 (1), 95--132. qu2012identification

    2012

  32. [40]

    The Review of Economic Studies, 84 (3), 1306--1345

    Qu, Zhongjun and Denis Tkachenko (2017), Global identification in dsge models allowing for indeterminacy. The Review of Economic Studies, 84 (3), 1306--1345. qu2017global

    2017

  33. [41]

    Journal of Monetary Economics, 52 (6), 1151--1166

    Rabanal, Pau and Juan F Rubio-Ram \' rez (2005), Comparing new keynesian models of the business cycle: A bayesian approach. Journal of Monetary Economics, 52 (6), 1151--1166. rabanal2005comparing

    2005

  34. [42]

    Econometrica, 80 (3), 1019--1061

    Ryan, Stephen P (2012), The costs of environmental regulation in a concentrated industry. Econometrica, 80 (3), 1019--1061. ryan2012costs

    2012

  35. [43]

    The Economic Journal, 127 (603), 1626--1664

    Santos, Carlos Daniel (2017), Sunk costs of r&d, trade and productivity: The moulds industry case. The Economic Journal, 127 (603), 1626--1664. santos2017sunk

    2017

  36. [44]

    Journal of Applied Econometrics, 15 (6), 645--670

    Schorfheide, Frank (2000), Loss function-based evaluation of dsge models. Journal of Applied Econometrics, 15 (6), 645--670. schorfheide2000loss

    2000

  37. [45]

    American economic review, 97 (3), 586--606

    Smets, Frank and Rafael Wouters (2007), Shocks and frictions in us business cycles: A bayesian dsge approach. American economic review, 97 (3), 586--606. smets2007shocks

    2007

  38. [46]

    International Economic Review, 54 (2), 495--523

    Suzuki, Junichi (2013), Land use regulation as a barrier to entry: evidence from the texas lodging industry. International Economic Review, 54 (2), 495--523. suzuki2013land

    2013

  39. [47]

    Econometrica, 81 (5), 1763--1803

    Sweeting, Andrew (2013), Dynamic product positioning in differentiated product markets: The effect of fees for musical performance rights on the commercial radio industry. Econometrica, 81 (5), 1763--1803. sweeting2013dynamic

    2013

  40. [48]

    International Journal of Industrial Organization, 61, 1--52

    Varela, Mauricio J (2018), The costs of growth: accelerated growth and crowd-out in the mexican supermarket industry. International Journal of Industrial Organization, 61, 1--52. varela2018costs

    2018

Pith tools

Reviewed June 25, 2026 · model on record in the stance chip above.