Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning LLMs on observational outcomes can teach spurious correlations, and DeconfoundLM removes observed confounders from rewards to recover causal text effects.

desk verdict Solid empirical cautions about observational fine-tuning, but the proposed DeconfoundLM method is under-specified and its simulation validates its own assumptions. read the letter →

arxiv 2506.00152 v1 pith:W232XZTJ submitted 2025-05-30 cs.LG econ.EMstat.ML

Erfan Loghmani This is my paper
classification cs.LGecon.EMstat.ML
keywords observationaldatacausalinferenceLLMalignmentconfoundingrewardmisspecificationdirectpreferenceoptimizationinstrumentalvariablesclick-throughrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the vast historical logs companies already collect can replace expensive A/B tests as supervision for aligning language models. It argues that naive fine-tuning on such observational outcomes teaches models to exploit spurious correlations, and it proposes DeconfoundLM, a fine-tuning procedure that estimates the contribution of observed confounders and subtracts it from the reward before reinforcement learning. Using a Stack Exchange case study, the paper shows that direct preference optimization amplifies a weekday artifact ("Happy Monday") from a 13.7% to a 21.2% generation rate. On Upworthy A/B headline data, models trained on observational inputs reach 0.74 ROC AUC versus 0.82 with experimental data, showing real but incomplete signal. In simulation, DeconfoundLM matches the sentiment gains of a true-reward baseline while suppressing confounder-driven team-name mentions, in both orthogonal and entangled confounding settings.

What carries the argument

The load-bearing object is the additive separable outcome model $y_i = g(T_i, \tilde{F}_i) + h(C_i) + \epsilon_i$, where $g$ captures the causal effect of the textual action and $h$ captures the effect of observed confounders. DeconfoundLM is a fine-tuning pipeline that first estimates $h$, usually through an instrumental-variable strategy, then subtracts it from the observed outcomes so that the reward model learns $g$ alone; the framework also allows Double Machine Learning or adversarial GMMs for the estimation step. The empirical demonstrations rest on three components: a two-stage SFT-plus-DPO setup on Stack Exchange answers, logistic and regression reward heads on Pythia embeddings for the Upworthy headline data, and a PPO loop on SmolLM2-360M with synthetic rewards generated as $s(T_i) + 0.1 p_i + \nu_i$ in both orthogonal and entangled confounding designs.

What would settle it

Generate a synthetic dataset with a non-separable outcome such as $y = g(T,F) \cdot h(C) + \epsilon$ while keeping everything else identical to the paper's MIND simulation; if DeconfoundLM-IV then fails to recover sentiment and continues to generate team-name mentions, the central claim that it removes the influence of observed confounders is falsified. Alternatively, run the method on an observational slice of a real A/B dataset where the true causal effect is known from the experiment and show that the deconfounded model ranks held-out pairs no better than naive fine-tuning.

Watch

Extended reading notes

Core claim

The central claim is that observational outcomes contain useful supervision for LLM alignment, but only if the confounders that shape those outcomes are explicitly removed from the reward signal. The paper demonstrates the failure mode on Academia Stack Exchange data: after SFT followed by DPO on historical answer scores, the model generates "Happy" in 21.2% of responses compared to 13.7% for SFT, because Monday-posted answers systematically score higher for engagement reasons unrelated to quality. The proposed remedy, DeconfoundLM, uses a separable outcome model $y = g(T,F) + h(C) + \epsilon$ and an instrumental-variable estimate of $h$ to build a deconfounded reward; in MIND-based simulations it recovers headline sentiment close to the noiseless true-reward baseline (0.842 and 0.850 across scenarios vs 0.881 for the noisy-observed baseline) while reducing team-name mentions that naive RL amplifies. The paper also establishes that regularization strength must grow with model size when training on observational data, and that tuning on validation loss alone picks a weaker regularizer than test performance requires.

Load-bearing premise

The method works only if the real-world outcome is additively separable into a text effect and a confounder effect, and if a valid instrument exists to estimate the confounder part; if either fails, the deconfounded reward is still biased.

Editorial extensions

If this is right

  • Observational logs alone give above-chance preference learning (0.74 ROC AUC on Upworthy pairs) but leave a stable gap versus experimental supervision (0.82), so causal data still matters when available.
  • Tuning hyperparameters by validation loss can under-regularize: the best validation MSE occurs at a smaller regularization than the best test ROC AUC, so practitioners should expect to regularize more aggressively than loss curves suggest.
  • Larger models need stronger regularization in observational settings; with fixed lambda, test performance becomes non-monotonic in model size.
  • DeconfoundLM prevents the reward model from encoding confounder-driven artifacts: it keeps positive reward-sentiment correlation in both orthogonal and entangled settings, where naive and feature-conditioned baselines go negative in the entangled case.
  • Naive fine-tuning on confounded observational data can actively worsen behavior, as when DPO raises spurious "Happy Monday" generation by 7.5 percentage points over SFT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the separability and instrument assumptions hold in practice, DeconfoundLM-type corrections could let firms substitute cheap historical logs for expensive experiments in content optimization, but only when the confounder is observed and a valid instrument exists.
  • A natural testable extension is to apply the deconfounding step to real A/B datasets with known confounder manipulations (e.g., time-of-day or audience segment) and compare held-out ranking AUC against naive fine-tuning; the paper's simulation evidence suggests the deconfounded model should win.
  • The entangled-confounding results hint that merely conditioning on the confounder (as input text or a layer feature) is insufficient; actively removing its additive contribution is what restores the reward-sentiment correlation.
  • The same additive-correction idea could be carried into fairness: if group membership enters $h(C_i)$, subtracting it before RL should reduce the model's tendency to replicate group-level outcome disparities, though the paper does not test this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies whether historical observational data can be used to fine-tune LLMs for content-optimization tasks, and it argues that naive fine-tuning on such data can learn spurious correlations. The authors first demonstrate this risk with a Stack Exchange 'Monday' experiment, where DPO amplifies a weekday marker, and with an Upworthy-based study comparing experimental and observational reward modeling. They then propose DECONFOUND LM, a method intended to estimate and remove the contribution of observed confounders from reward signals using an instrumental-variable strategy, and they evaluate it in synthetic headline-generation experiments built on the MIND dataset. The claimed contributions are that observational signals are valuable but risky, that stronger regularization is needed, and that DECONFOUND LM recovers causally relevant reward components.

Significance. The problem addressed is practically important: many organizations have abundant observational logs but cannot run controlled experiments, and the paper correctly identifies a real failure mode in which fine-tuning amplifies spurious temporal or engagement-related signals. The Monday experiment is a strength: it is carefully constructed with multiple seeds, reports standard errors, and uses appropriate t-tests, giving clean evidence of spurious-signal amplification. The Upworthy analysis is also transparent about data processing and regularization choices. If DECONFOUND LM were fully specified and its simulation evidence were non-circular, this would be a useful contribution to causal alignment for LLMs. As it stands, however, the central method is underspecified and the simulation validation is self-confirming, so the paper's main claim about confounder removal is not yet established. The paper is honest in Appendix E about the separability limitation, but that honesty does not repair the gap between the contribution statement in Section 1 and what is actually demonstrated.

major comments (4)
  1. [4 (Proposed method)] Section 4 states that 'we apply an instrumental variable strategy to estimate the confounding component,' but it never defines the instrument, the first-stage regression, the exclusion restriction, or the estimating equations. Without these elements, the DECONFOUND LM-IV rows in Table 2 and Table 5 are not reproducible and no consistency or identification argument can be checked. This is the core of the method claim in Section 1, so the specification must be provided explicitly.
  2. [4.1, Eqs. (4)-(6)] The simulation outcome y_i = s(T_i) + 0.1 p_i + nu_i is exactly the additively separable form y = g(T,F) + h(C) + eps assumed in Eq. (2), with p_i observed and additive. In the entangled scenario, Eq. (6) makes p_i a linear function of s(T_i), but the outcome remains additive in s(T_i) and p_i, so the assumed separability is never violated. The experiments therefore confirm that the estimator fits the model that generated the data; they provide no evidence for non-separable outcomes, misspecified confounders, or invalid instruments. The unqualified claim in Section 1 that DECONFOUND LM 'removes the influence of observed confounders' is not supported beyond this self-confirming setting. Appendix E concedes that a more general framework is future work; this limitation should be reflected in the main-text claims as well.
  3. [Appendix E, Eq. (7)] The framework assumes E[epsilon_i | X_i] = 0. Since X_i includes the observed confounder C_i, this exogeneity condition already permits consistent estimation of g by directly conditioning on C_i, and an instrumental-variable strategy is not needed. If the intended setting is instead one of endogeneity (E[epsilon | X] != 0), then the IV approach requires a valid instrument satisfying relevance and exclusion restrictions, none of which is stated. These two identification statements are in tension and must be reconciled for the method to be well defined.
  4. [1 and 4.1] The contribution list claims that DECONFOUND LM 'consistently improves model behavior,' but the only evaluation of the method is the synthetic MIND sports-headline setting. The two real-data studies (Stack Exchange and Upworthy) demonstrate the problem of spurious correlations but do not evaluate the proposed correction. A real-data application or, at minimum, a semisynthetic benchmark with realistic observed confounders is needed before the 'consistently' claim can be assessed.
minor comments (5)
  1. [3.1] The main text says the observational dataset keeps 'just one headline package' from each A/B test; Appendix C clarifies that this is applied only to the training tests. Please state this in the main text to avoid ambiguity about the validation and test splits.
  2. [3.2, Figure 4d] The 'Monthly correlation' metric is described qualitatively in the text; please define it precisely (e.g., an equation for the correlation between predicted monthly CTRs and observed monthly CTRs) so that the reader can interpret the figure without guessing.
  3. [Abstract] The abstract refers to a project page with code and resources, but no URL or code appendix is provided in the manuscript. Please include a link or a supplementary code release statement.
  4. [Table 2 caption] The caption says 'the first four models are identical across both scenarios'; this should say 'the first four rows' or 'the first four model variants' to avoid confusion with the six evaluated approaches.
  5. [Appendix D] The hyperparameter selection for reward models is described using train and eval loss curves; please clarify whether the same reward-validation split is used for selecting the L2 regularization lambda in Section 3.2 and for the simulation reward models.

Circularity Check

1 steps flagged · score 6.0 of 10

DeconfoundLM's central evidence is partially circular: the simulation outcome y_i = s(T_i) + 0.1 p_i + nu_i is exactly the separable model assumed in Eq. (2), and the IV estimator is never specified.

  1. self definitional [Section 4.1, Eq. (4) vs. Section 4, Eq. (2); Appendix E]
    ""In both scenarios, we model the outcome as: yi = s(Ti) + 0.1 pi + νi" (Eq. 4). This is exactly the decomposition assumed in Eq. (2): "yi = g(Ti, F~i) + h(Ci) + ϵi", with h(Ci) = 0.1 pi."

    The simulation's data-generating process is literally the paper's own assumed separable form: the target effect g is s(T_i), the confounder component h(C_i) is 0.1 p_i, and the noise is ν_i. The entangled case in Eq. (6) only modifies how p_i is constructed; the outcome y_i remains additive in s(T_i) and p_i, so separability is never violated. Therefore the reported success of DECONFOUND LM-IV in Table 2 shows only that an estimator of h(C) can invert the generative equation the authors wrote down under their own assumption. It is not evidence that the method identifies h(C) when the model is misspecified or when no valid instrument exists.

full rationale

The paper contains no load-bearing self-citations: all cited works are external, and the author does not invoke a prior result of his own to force the method. The Monday experiment and the Upworthy observational-vs-experimental comparison are independent empirical demonstrations that observational fine-tuning can internalize spurious signals; these are not circular. The partial circularity lies in the central claim about DECONFOUND LM. The method's identification is never written down: Section 4 merely states that an instrumental variable strategy is applied, without defining the instrument, first stage, or exclusion restriction. The only controlled evaluation of the method is the simulation, whose outcome equation y_i = s(T_i) + 0.1 p_i + ν_i is exactly the separable g+h model assumed in Eq. (2), with the confounder component equal to the inserted 0.1 p_i. Thus the simulated 'recovery' of causal relationships reduces, by construction, to inverting the generative model that the authors themselves defined; it cannot validate the method against model misspecification or invalid-instrument settings. The Appendix E acknowledgement of the separability assumption is honest, but it undercuts the unqualified contribution statement in Section 1 that DECONFOUND LM 'removes the influence of observed confounders from the reward signal.' Overall, this is a partial circularity: independent empirical content exists, but the central method's validation rests on a self-defined generative model plus an unspecified estimator.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; DeconfoundLM is a procedure, not an entity. The method validation rests on two hand-built simulation DGPs, an under-specified IV, and the acknowledged additive-separability and exogeneity assumptions. The real-world analyses do not test DeconfoundLM at all, so the ledger is dominated by the simulation design, which mirrors the paper's own model by construction.

free parameters (2)
  • L2 regularization weight lambda (Upworthy reward models) = 18,000 validation-optimal; 50,000 test-optimal for Pythia-12B
    Central to the claim that observational fine-tuning needs stronger regularization. The optimal value is selected on data, and the gap between validation-optimal and test-optimal lambda is a fitted observation, not a predicted effect.
  • Simulation DGP constants: confounder weight 0.1; team popularity weights 1, 2, 3; entangled coefficient -10.5 = 0.1, 1, 2, 3, -10.5 (Eqs. 4-6)
    Chosen by hand to create the orthogonal and entangled scenarios. The small 0.1 weight keeps sentiment dominant in the outcome, and the -10.5 coefficient creates the entangled failure mode; these constants determine the difficulty of the task and condition all comparative results.
assumptions (5)
  • ad hoc to paper Additive separability of the outcome: y = g(T,F) + h(C) + eps (Eq. 2)
    The method's identification strategy assumes the confounding contribution h(C) is additively separable and can be subtracted. Appendix E acknowledges "we assume a separable functional form in the final line of Equation 7" and states a more general framework is future work.
  • domain assumption Exogeneity E[eps|X] = 0 (Appendix E)
    Standard econometric assumption, invoked in Appendix E with citations to Zou and Zhang (2009) and Fan and Liao (2014), while acknowledging that incidental endogeneity can arise from selection bias or misspecification.
  • ad hoc to paper Validity of the unspecified instrumental variable strategy (Section 4)
    The paper states "we apply an instrumental variable strategy to estimate the confounding component" but never defines the instrument, its relevance condition, or the exclusion restriction. The entire DeconfoundLM simulation rests on this unspecified procedure.
  • ad hoc to paper Sentiment s(T_i) as the true causal quality of a headline (Eq. 4)
    The simulation defines quality as the distilbert sentiment score of the headline, and the evaluation then measures sentiment recovery, so the metric and the target coincide, which inflates the apparent success of methods that track sentiment.
  • domain assumption Weekday engagement is a confounder, not a causal effect, in the Monday experiment
    The paper infers confounding from parallel trends in answer scores and question scores and views (Figure 1). The interpretation is plausible but not identified with certainty; the paper itself notes in Section 4.1 that for real data "we cannot determine whether these factors have purely confounding or partially causal effects."

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective." pith.science (2026). https://pith.science/paper/W232XZTJ

@misc{pith2026250600152,
  author       = {Pith},
  title        = {Pith review of: Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W232XZTJ}},
  note         = {Machine review of arXiv:2506.00152}
}
read the original abstract

Large language models are being widely used across industries to generate content that contributes directly to key performance metrics, such as conversion rates. Pretrained models, however, often fall short when it comes to aligning with human preferences or optimizing for business objectives. As a result, fine-tuning with good-quality labeled data is essential to guide models to generate content that achieves better results. Controlled experiments, like A/B tests, can provide such data, but they are often expensive and come with significant engineering and logistical challenges. Meanwhile, companies have access to a vast amount of historical (observational) data that remains underutilized. In this work, we study the challenges and opportunities of fine-tuning LLMs using observational data. We show that while observational outcomes can provide valuable supervision, directly fine-tuning models on such data can lead them to learn spurious correlations. We present empirical evidence of this issue using various real-world datasets and propose DeconfoundLM, a method that explicitly removes the effect of known confounders from reward signals. Using simulation experiments, we demonstrate that DeconfoundLM improves the recovery of causal relationships and mitigates failure modes found in fine-tuning methods that ignore or naively incorporate confounding variables. Our findings highlight that while observational data presents risks, with the right causal corrections, it can be a powerful source of signal for LLM alignment. Please refer to the project page for code and related resources.

Figures

Figures reproduced from arXiv: 2506.00152 by the authors.

Figure 1
Figure 1. Weekly temporal patterns in Stack Exchange engagement. User scores and views exhibit [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Temporal patterns in user engagement. The left plot shows variation in average click [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. ROC on held-out Upworthy headline pairs. (a) ROC curves for models trained with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Effect of regularization strength (λ) on different evaluation metrics for the Pythia-12B model. Temporal pattern overfitting. As discussed in Section 3.1, temporal variation in CTRs is a potential confounder in observational data. To assess how much models internalize …
Figure 5
Figure 5. Figure 5: Regularization dynamics across model sizes. (a) Models require larger regularization to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Loss curves for reward model training and evaluation across different learning rates for [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Loss curves for reward model training and evaluation across different learning rates for the [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Loss curves for reward model training and evaluation across different learning rates for the [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 23 canonical work pages

  1. [1]

    How ai outperforms humans at creative idea generation

    Noah Castelo, Zsolt Katona, Peiyao Li, and Miklos Sarvary. How ai outperforms humans at creative idea generation. Available at SSRN 4751779, 2024

    work page 2024

  2. [2]

    Ai–human hybrids for marketing research: Leveraging large language models (llms) as collaborators

    Neeraj Arora, Ishita Chakraborty, and Yohei Nishimura. Ai–human hybrids for marketing research: Leveraging large language models (llms) as collaborators. Journal of Marketing, 89 (2):43–70, 2025

    work page 2025

  3. [3]

    Frontiers: Can large language models capture human prefer- ences? Marketing Science, 43(4):709–722, 2024

    Ali Goli and Amandeep Singh. Frontiers: Can large language models capture human prefer- ences? Marketing Science, 43(4):709–722, 2024

    work page 2024

  4. [4]

    Lola: Llm-assisted online learning algorithm for content experiments

    Zikun Ye, Hema Yoganarasimhan, and Yufeng Zheng. Lola: Llm-assisted online learning algorithm for content experiments. arXiv preprint arXiv:2406.02611, 2024

    arXiv 2024

  5. [5]

    How reliable is human feedback for aligning large language models? arXiv preprint arXiv:2410.01957, 2024

    Min-Hsuan Yeh, Leitian Tao, Jeffrey Wang, Xuefeng Du, and Yixuan Li. How reliable is human feedback for aligning large language models? arXiv preprint arXiv:2410.01957, 2024

    arXiv 2024

  6. [6]

    A/b testing: A systematic literature review

    Federico Quin, Danny Weyns, Matthias Galster, and Camila Costa Silva. A/b testing: A systematic literature review. Journal of Systems and Software, page 112011, 2024. 10

    work page 2024

  7. [7]

    Causal alignment: Augmenting language models with a/b tests

    Panagiotis Angelopoulos, Kevin Lee, and Sanjog Misra. Causal alignment: Augmenting language models with a/b tests. Available at SSRN, 2024

    2024

  8. [8]

    Using advanced llms to enhance smaller llms: An interpretable knowledge distillation approach

    Tong Wang, K Sudhir, and Dat Hong. Using advanced llms to enhance smaller llms: An interpretable knowledge distillation approach. arXiv preprint arXiv:2408.07238, 2024

    work page arXiv 2024

Show all 43 references
  1. [9]

    Test & roll: Profit-maximizing a/b tests

    Elea McDonnell Feit and Ron Berman. Test & roll: Profit-maximizing a/b tests. Marketing Science, 38(6):1038–1058, 2019

    2019

  2. [10]

    An empirical meta-analysis of e-commerce a/b testing strategies

    Alex P Miller and Kartik Hosanagar. An empirical meta-analysis of e-commerce a/b testing strategies. The Wharton School, University of Pennsylvania, 2020

    2020

  3. [11]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  4. [12]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [13]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  6. [14]

    The importance of human-labeled data in the era of llms

    Yang Liu. The importance of human-labeled data in the era of llms. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 7026–7032, 2023

    2023

  7. [15]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  8. [16]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37:126207–126242, 2024

    2024

  9. [17]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  10. [18]

    RLAIF: Scal- ing reinforcement learning from human feedback with AI feedback, 2024

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scal- ing reinforcement learning from human feedback with AI feedback, 2024. URL https: //openreview.net/forum?id=A...

    2024

  11. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  12. [20]

    A general language assistant as a laboratory for alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    2021 arXiv

  13. [21]

    The upworthy research archive, a time series of 32,487 experiments in us media

    J Nathan Matias, Kevin Munger, Marianne Aubin Le Quere, and Charles Ebersole. The upworthy research archive, a time series of 32,487 experiments in us media. Scientific Data, 8 (1):195, 2021

    2021

  14. [22]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023

  15. [23]

    Double/debiased machine learning for treatment and structural parameters

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68, 2018

    2018

  16. [24]

    Minimax estimation of conditional moment models

    Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems , 33: 12248–12262, 2020. 11

    2020

  17. [25]

    Mind: A large-scale dataset for news recommendation

    Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th annual meeting of the association for computational linguistics , pag...

    2020

  18. [26]

    Generative brand choice

    Kevin Lee. Generative brand choice. Technical report, Working Paper, 2024

    2024

  19. [27]

    Deep neural networks for estimation and inference

    Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):181–213, 2021

    2021

  20. [28]

    Deep generalized method of moments for instrumental variable analysis

    Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. Advances in neural information processing systems , 32, 2019

    2019

  21. [29]

    Causal regressions for unstructured data

    Amandeep Singh and Bolong Zheng. Causal regressions for unstructured data. In Causal Representation Learning Workshop at NeurIPS 2023, 2023. URLhttps://openreview.net/ forum?id=Zs3C7zytfp

    2023

  22. [30]

    Secrets of rlhf in large language models part i: Ppo

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023

    2023 arXiv

  23. [31]

    Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. arXiv preprint arXiv:2406.09279, 2024

    2024 arXiv

  24. [32]

    Bias in data-driven artificial intelligence systems—an introductory survey

    Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria- Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. Bias in data-driven artificial intelligence systems—an introductory survey. Wiley Int...

    2020

  25. [33]

    Causal confusion and reward misidentification in preference-based reward learning

    Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601, 2022

    2022 arXiv

  26. [34]

    Sycophancy to subterfuge: Investigating reward-tampering in large language models

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162, 2024

    2024 arXiv

  27. [35]

    Odin: Disentangled reward mitigates hacking in rlhf

    Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319, 2024

    2024 arXiv

  28. [36]

    Beyond reward hacking: Causal rewards for large language model alignment

    Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501.09620, 2025

    2025 arXiv

  29. [37]

    Smollm2: When smol goes big – data-centric training of a small language model, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíˇcek, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo ...

    2025 arXiv

  30. [38]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022

    2022

  31. [39]

    distilbert-base-multilingual-cased-sentiments-student (re- vision 2e33845), 2023

    Lik Xun Yuan. distilbert-base-multilingual-cased-sentiments-student (re- vision 2e33845), 2023. URL https://huggingface.co/lxyuan/ distilbert-base-multilingual-cased-sentiments-student

    2023

  32. [40]

    A survey on bias and fairness in machine learning

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1–35, 2021

    2021

  33. [41]

    On the adaptive elastic-net with a diverging number of parameters

    Hui Zou and Hao Helen Zhang. On the adaptive elastic-net with a diverging number of parameters. Annals of statistics, 37(4):1733, 2009. 12

    2009

  34. [42]

    Endogeneity in high dimensions

    Jianqing Fan and Yuan Liao. Endogeneity in high dimensions. Annals of statistics, 42(3):872, 2014. 13 A Related work Our research intersects three core areas: (1) the use of language models in business and marketing applications, (2) econometric approaches to causal inference ...

    2014

  35. [43]

    packages

    propose a distillation framework in which smaller models learn response behaviors from larger LLMs, assuming that the teacher model’s outputs serve as a reliable proxy for optimal performance. In a different application area, Lee [26] addresses demand prediction for new produc...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.