How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation

Hao Yang; Lei Li; Lingyi Meng; Mengda Yu; Qinghua Zhao

REVIEW 5 major objections 5 minor 2 references

How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chain-of-thought prompting improves reasoning by pruning the decoding space toward an answer template, with higher template adherence correlating with better performance across six models and nine datasets.

desk verdict Plausible framing, but the headline decoding-space evidence rests on an under-specified, likely confounded metric; the activation-side sign flip is the more independent and keep-worthy result. read the letter →

arxiv 2507.20758 v2 pith:ZOTJMI3N submitted 2025-07-28 cs.AI

Hao Yang , Qinghua Zhao , Lei Li , Lingyi Meng , Mengda Yu This is my paper

classification cs.AI

keywords chain-of-thoughtpromptingmechanisticinterpretabilitydecodingspacepruninganswertemplateadherenceneuronactivationinformationflowlargelanguagemodelspromptengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why chain-of-thought (CoT) prompting improves reasoning in large language models. It argues that CoT works as a decoding-space pruner: the prompt's step-by-step exemplars teach the model an answer template such as $E_p \xrightarrow{O} E_g + S_l$, input entities, an operation, a derived entity, and a final 'so the answer is...' statement, and the model then imitates that template while filling in task content from the question. The evidence is a strong correlation between how faithfully a model adheres to this structure and how often it answers correctly, together with probability distributions that become more concentrated and lower-entropy under CoT. At the neuron level the paper reports that CoT reduces the number of activated neurons in open-domain tasks but increases it in closed-domain tasks, with the differences concentrated in the final third of the layers. If this is right, prompt design should focus on the reasoning structure rather than on the correctness of the intermediate steps.

What carries the argument

The load-bearing instrument is a reverse information-flow trace through three stages: decoding, projection, and activation. Its central object is the reasoning structure $E_p \xrightarrow{O} E_g + S_l$, where $E_p$ are input entities, $O$ is the operation or predicate, $E_g$ is a derived intermediate entity, and $S_l$ is the final answer statement; adherence to this structure is scored by an Imitation Count that requires entity extraction, detection of intermediate reasoning steps, and a terminal 'the answer is' phrase. At the projection stage the machinery is the probability density and entropy of generated tokens, and at the activation stage it is the count of neurons with positive output in feed-forward layers. These measurements are what carry the pruning claim: the same structure that predicts accuracy also explains the concentrated probabilities and the task-dependent neuron activation.

What would settle it

Rewrite the intermediate steps of CoT outputs on GSM8K so the template phrases and the entity-operation sequence are preserved but the arithmetic is wrong; if accuracy stays near the original CoT level, the pruning mechanism is supported, whereas if accuracy collapses, the correlation in Figure 2 is driven by reasoning content rather than by template adherence.

Watch

Extended reading notes

Core claim

The central claim is that CoT prompting constrains the decoding space by exploiting answer templates rather than by making the model reason more logically. The paper introduces a CoT reasoning structure $E_p \xrightarrow{O} E_g + S_l$ and measures adherence to it with an Imitation Count built from keyword patterns and the terminal phrase 'the answer is'. On GSM8K, across five models, this count correlates with accuracy at $r=0.75$ to $0.92$, with $R^2$ values from $0.57$ to $0.84$, so template adherence alone explains a substantial portion of performance variance. The projection analysis shows that CoT yields higher and more peaked token probabilities and lower vocabulary entropy, consistent with a narrowed prediction set. The activation analysis shows that CoT lowers average neuron activation for open-domain tasks and raises it for closed-domain tasks, with the strongest effects in the final third of layers; the paper interprets this as CoT acting as a 'pruner' where the answer space is open and an 'amplifier' where it is closed.

Load-bearing premise

The central claim rests on the assumption that the hand-built Imitation Count, with its keyword lists and the process-verb threshold of four, actually measures adherence to the CoT reasoning structure rather than lexical fluency, answer correctness, or generation length; the activation and entropy comparisons further assume that averaging over generation steps does not confound CoT with standard prompts even though CoT outputs are much longer.

Editorial extensions

If this is right

CoT prompts can be selected or adapted by matching the target task's reasoning structure; cross-dataset transfer results on GSM8K show that structurally aligned prompts raise both adherence and accuracy.
Because CoT produces more concentrated token probabilities and lower entropy at the answer step, decoding-time confidence estimates and early-exit or abstention decisions become more reliable under CoT.
The concentration of activation differences in the final third of layers implies that future interventions, such as targeted activation patching or pruning, should focus there.
The task-dependent modulation of neuron activation means a single fixed intervention will not transfer across open-domain and closed-domain tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If template adherence is what carries CoT's benefit, then the intermediate reasoning tokens may be largely interchangeable: replacing the chain with a structured placeholder that reproduces the same decoding constraints should preserve much of the gain, yielding a cheaper 'hidden CoT' variant.
The pruning view predicts that prompt-optimization by template matching will outperform content-rich exemplar selection on unseen tasks, a claim that can be tested without any internal-state measurement.
The layer-specific activation differences suggest a causal experiment the paper did not run: suppressing later-layer neurons in open-domain tasks and boosting them in closed-domain tasks should respectively improve and degrade accuracy if CoT's dual role is real.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper has a real empirical core, but the headline mechanism is not supported by the headline figure. The Imitation Count metric is the load-bearing piece and it is under-specified; I think the stress-test is right that it conflates output length and generic lexical fluency with structural adherence. The activation-side findings are more independent and are the most interesting thing here.

What is actually new: a quantified correlation between prompt-transfer adherence and GSM8K accuracy across five models, a consistent sign flip in layer-wise FFN activation differences between open- and closed-domain tasks, and entropy evidence that CoT concentrates output distributions. The "format over content" hypothesis is already in Madaan et al. 2023, Schaeffer et al. 2023, and Li et al. 2025, so the conceptual novelty is modest, but nobody had traced the effect through decoding, projection, and activation with this many models. The paper also releases code and data, which is real and useful.

The soft spots are in the decoding phase. Appendix E describes a three-condition binary tag, but Figure 2 reports an Imitation Count from 0 to 1500 with no counting formula. The keyword lists in Table 5 are generic connectives and process words; longer outputs will contain more of them almost mechanically. With six points per model, the strong r values could just be tracking how verbose or helpful each transferred prompt is, not template-driven pruning. The authors' own Limitations paragraph concedes the findings are heuristic rather than causal, which is the right reading. The circularity worry is fair for Figure 2 specifically, but not for the whole paper: the entropy and activation analyses do not depend on the template.

The activation results are the better evidence. The open/closed sign flip is visible across three model families and several datasets, and it is less vulnerable to the length confound because it is per-token averaged. Still, there are no significance tests, no control for answer-space size, and averaging over generation steps could hide position-dependent effects. The sign flip is worth reporting as an observation, not as a demonstrated mechanism.

Who this is for: people working on CoT interpretability and prompt engineering. It deserves a serious referee. A revision should validate the Imitation Count (or replace it), add statistical tests, and control for output length and correctness. If those are fixed, the activation pattern alone is citable.

Referee Report

5 major / 5 minor

Summary. The paper reports a mechanistic interpretability study of vanilla chain-of-thought (CoT) prompting across six LLMs and nine datasets, tracing information flow through decoding (generated text), projection (token probabilities and entropy), and activation (FFN neuron counts). The central proposal is that CoT acts as a decoding-space pruner by inducing adherence to an abstract template E_p -> O -> E_g + S_l, supported by correlations between a hand-built "Imitation Count" and GSM8K accuracy under prompt transfer. The paper also reports that CoT concentrates token-level probability distributions and modulates neuron activation in a task-dependent manner (lower in open-domain, higher in closed-domain tasks). Code and data are released.

Significance. The template-adherence mechanism is a plausible and practically relevant hypothesis, and if the correlation were robust, it would strengthen the 'format imitation' line of work on CoT. The entropy and activation analyses are independent of the hand-built template metric and provide complementary, non-circular evidence that CoT changes internal processing. The authors are candid in the Limitations section that the findings are heuristic rather than conclusively causal, and the release of code and data is a strength. However, the central quantitative claim currently rests on an undefined, length-sensitive metric and a very small number of hand-picked prompt conditions, so the significance of the result cannot be fully assessed from the manuscript as written.

major comments (5)

[§4.1, Appendix E, Figure 2] The 'Imitation Count' used as the x-axis in Figure 2 is never defined. Appendix E describes only a binary three-condition adherence tag (entities extracted, intermediate steps present, final 'the answer is' phrase), with no formula or algorithmic description connecting that tag to the 0-1500 integer counts plotted in Figure 2. Without the counting definition, the correlations cannot be reproduced or interpreted. Moreover, because the keyword lists in Table 5 contain generic connectives and process words ('then', 'so', 'originally', '+', 'is'), the count will mechanically increase with output length; transferred prompts that elicit more verbose generations will therefore score higher regardless of whether they instantiate the claimed reasoning structure. The authors should provide the exact counting formula, release the scoring code, and report a length-controlled or per-token-normalized version of the analysis.
[§4.1 (Figure 2)] The statistical support for the central claim is weak as presented. Each model contributes only n=6 points, and those points are not independent samples but a hand-picked set of prompts (the native CoT prompt and five transferred prompts). The text states that 'four out of five models' reach significance, but Figure 2 shows four panels and Section 3.1 lists six models; the model count and the set of models used for the correlation analysis need to be reconciled. With n=6, a single prompt choice can drive r, and no multiple-comparison correction or permutation test is reported. The authors should report all models, use an appropriate test that treats prompts as a sample, and show that the correlation is not an artifact of prompt verbosity or prior prompt quality.
[§4.2 (Figures 3–4)] The entropy and token-probability comparisons do not control for the large difference in generation length and token position between CoT and standard prompts. CoT outputs are much longer, so averaging over all generation steps mixes different positions, and the 'answer is' phrase occurs at very different token indices in the two conditions. Figures 3 and 4 show density and scatter plots with no significance tests, confidence intervals, or effect sizes, and the sample size (the figures index only 50 test samples) is not stated in the text. A position-matched or length-matched comparison, together with a paired test across prompts or models, is needed before claiming that CoT 'sharpens decision boundaries'.
[§4.3 (Figures 5–6)] The neuron-activation claims are based on averages over all generated tokens (A_bar = (1/T) sum_t A_t), which conflates activation differences with differences in sequence length and token type between CoT and standard generations. The open-domain versus closed-domain pattern (negative late-layer differences for GSM8K/Bamboogle, positive for Coin Flip/AQuA/Sports) is asserted from visual inspection of Figure 6 and Appendix I without significance testing or a precise definition of the partition (e.g., where Date and Last Letters fall). The authors should provide per-position or length-matched activation curves, report statistical tests over models and datasets, and specify the open/closed classification criterion before this task-dependent claim can be accepted.
[§4.1 and Limitations] The reasoning-structure analysis is partly self-referential. The Imitation Count is constructed from the authors' own template (E_p -> O -> E_g + S_l) and keyword lists, and then the correlation between that count and accuracy is presented as evidence that the template drives CoT's benefit. This does not validate the template independently. The entropy and activation analyses are independent and therefore valuable, but they do not directly test the pruning mechanism. The Limitation section's concession that the correlations are 'heuristic or suggestive rather than strictly conclusive' is an accurate description, but the abstract and Section 4.1 state the template conclusion more strongly than the evidence supports. The authors should either validate the metric against human annotation or an independent structural measure, or soften the causal framing accordingly.

minor comments (5)

[Table 1 and Appendix A] There are several typos: 'destiny' should be 'destination' in the AQuA example, 'SV AMP' has inconsistent spacing, and Table 3 duplicates Table 1 in the appendix.
[Appendix G and figures] The heading 'Kernal density' should be 'Kernel density', and figure captions contain tokenization artifacts such as 'T oken Index' and 'Con caternating'; these should be cleaned.
[Abstract and Section 1] The abstract gives an anonymous OpenReview link (anonymous.4open.science) while the footnote gives a GitHub URL; please unify the availability statement.
[References] Several references are incomplete or informal, for example 'bench authors' should be 'BIG-Bench authors', and multiple entries use 'and 1 others' instead of full author lists.
[Section 4.3] The notation for the FFN equation is inconsistent: the activation function argument is written as 'Act(˜h(l)W(l)up )' in the text and as 'Act(˜h(l)_t, W(l)_up)_j' in the formula for A_t; please harmonize the notation.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the central correlations and activation measurements are independent empirical evidence, with one peripheral self-citation.

full rationale

The paper's main quantitative claims are empirical measurements: CoT vs standard token probabilities, entropy, and FFN activation counts are direct model outputs compared under two prompt conditions, and the Figure 2 correlation between Imitation Count and GSM8K accuracy is a computed Pearson r over six transferred-prompt points. None of these quantities is fitted to the target claim or defined as the target claim. The reasoning structure E_p -> O -> E_g + S_l (Eq. 1) is operationalized through the same test-point keyword categories used to compute Imitation Count (Appendix B, E), so the correlation does not independently validate that taxonomy as the causal mechanism; this is a construct-validity limitation, not a circular reduction, because the correlation could have failed and is not forced by the metric. The authors' Limitations section explicitly concedes that the observed correlations are heuristic rather than causal, which further confirms the claims are presented as evidence rather than as derivations from their assumptions. The only self-citation is Li et al. (2024), co-authored by Qinghua Zhao, used to support the secondary observation that correct answers have lower entropy; it is not load-bearing for the central pruning claim. The projection and activation analyses are independent of the template metric and provide non-circular evidence that CoT changes internal processing. Overall, no prediction in the paper reduces to its inputs by construction, so the circularity score is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The headline claims rest on two hand-built instruments: the test-point keyword lists and the CoT Reasoning Structure template. Neither is validated against an external benchmark, and the activation analysis depends on a binary positivity threshold and an assumed decision-point phrase. The paper's own Limitations acknowledge the heuristic nature of the conclusions.

free parameters (2)

process-verb count threshold = 4
Appendix E uses a count exceeding four process verbs (flips, is, was, are, be, were) as an indicator of reasoning activity for Coin Flip and Last Letter tasks; this threshold is hand-chosen.
neuron activation positivity threshold = 0
A neuron is counted as activated if Act() > 0, following Geva et al. (2021b); the paper adopts this binary threshold without testing alternatives.

assumptions (4)

domain assumption FFN neurons with positive post-activation values constitute 'activated' neurons whose count reflects task-relevant engagement.
The paper equates positive FFN outputs with 'engaged' neurons and uses the count as a measure of processing focus (Section 4.3), following prior work; if the threshold is arbitrary, the activation claims weaken.
domain assumption The phrase 'the answer is' is a reliable decision point for comparing CoT and standard generation probability.
Section 4.2 uses this phrase across all datasets without validating that it is the decision point in both prompt conditions.
ad hoc to paper The CoT reasoning structure E_p -> O -> E_g + S_l is a faithful formalization of CoT reasoning, and the Imitation Count measures adherence to it.
Introduced in Section 4.1; the empirical correlation depends entirely on this metric.
ad hoc to paper Datasets can be partitioned into open-domain and closed-domain based on answer format, and this partition explains the sign of late-layer activation differences.
Section 4.3; the paper does not test alternative partitions or confounds.

invented entities (2)

CoT Reasoning Structure (E_p -> O -> E_g + S_l)
purpose: Formal template used to compute Imitation Count and support the decoding-space-pruner hypothesis.
Descriptive schema introduced in Section 4.1; no falsifiable prediction outside the paper.
Test point categories (time, action, loc&peo, number)
purpose: Keyword categories used to measure imitation from prompts and questions.
Hand-constructed lists in Appendix B; their relevance is asserted, not externally validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation." pith.science (2026). https://pith.science/paper/ZOTJMI3N

@misc{pith2026250720758,
  author       = {Pith},
  title        = {Pith review of: How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZOTJMI3N}},
  note         = {Machine review of arXiv:2507.20758}
}

read the original abstract

Chain-of-Thought (CoT) prompting significantly enhances model reasoning, yet its internal mechanisms remain poorly understood. We analyze CoT's operational principles by reversely tracing information flow across decoding, projection, and activation phases. Our quantitative analysis suggests that CoT may serve as a decoding space pruner, leveraging answer templates to guide output generation, with higher template adherence strongly correlating with improved performance. Furthermore, we surprisingly find that CoT modulates neuron engagement in a task-dependent manner: reducing neuron activation in open-domain tasks, yet increasing it in closed-domain scenarios. These findings offer a novel mechanistic interpretability framework and critical insights for enabling targeted CoT interventions to design more efficient and robust prompts. We released our code and data at https://anonymous.4open.science/r/cot-D247.

Figures

Figures reproduced from arXiv: 2507.20758 by the authors.

**Figure 1.** Keyword imitation results (Gemma2-27B is reported). dataset X were applied to dataset Y , and we evaluated changes in generated content characteristics. Analysis [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Correlation between reasoning structure adherence and task accuracy on the GSM8K dataset. Pearson correlation coefficients (r), p-values, and R2 scores are reported for each model. performance variance is explained by structural adherence alone. Scatter plots further reveal differences between prompt types. Prompts whose reasoning structures align with GSM8K (e.g., arithmetic or sequential patterns) induce higher i… view at source ↗

**Figure 3.** Kernel density estimation (Gemma2-9B is reported, see Appendix G for more details). 0 10 20 30 40 50 Index of test samples 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Logits entropy COT Correct COT Wrong Standard Correct Standard Wrong (a) AQuA 0 10 20 30 40 50 Index of test samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Logits entropy COT Correct COT Wrong Standard Correct Standard Wrong (b) Sports 0 10 20 30 40 50 Index of test … view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Entropy (Gemma2-27B is reported, see Appendix H for more details) swer is ...” across all datasets and prompt types. We select this phrase as it marks the final decision point, revealing CoT’s impact on decision certainty. We model the probability sequence as P = [p(“…

**Figure 5.** Figure 5: Average Neuron Activation Count (Full results are in Appendix J). counts, while that for other datasets (e.g. GSM8K) is “tall and narrow”, suggesting a tighter cluster around the mean. These differences likely reflect the varying activation patterns associated with dif…

**Figure 6.** Figure 6: Layer-wise activation differences across layers for models (3B, 8B, 70B) and datasets (GSM8K and Coin Flip). For full datasets results, please refer to Appendix I. of CoT’s impact in later layers may support that CoT’s guidance is most relevant during the structuring …

**Figure 7.** Figure 7: Results of Gemma2-2b [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Results of Gemma2-9b. 0.00 0.02 0.04 0.06 Proportion Prompt time action loc&peo number gsm8k svamp bamboogle sports date strategyqa coinflip lastletters 0.00 0.02 0.04 0.06 Proportion Question (a) AQuA 0.00 0.05 0.10 Proportion Prompt time action loc&peo number aqua sv…

**Figure 9.** Figure 9: Results of LLaMA2-13b [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Probability value of each generated token (results of Gemma2-2b is shown). [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Probability value of each generated token (the results of LLaMA2-13b is shown). [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

**Figure 12.** Figure 12: Probability value of each generated token (the results of Gemma2-27b is shown). [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

**Figure 13.** Figure 13: Kernal density of Gemma2-2B. 0.0 0.2 0.4 0.6 0.8 1.0 Probability of generated token 0 1 2 3 4 5 6 7 Density AQuA Standard CoT 0.0 0.2 0.4 0.6 0.8 1.0 Probability of generated token 0 2 4 6 8 10 12 Density GSM8K Standard CoT 0.0 0.2 0.4 0.6 0.8 1.0 Probability of gener…

**Figure 14.** Figure 14: Kernal density of LLaMA2-13B. 0.0 0.2 0.4 0.6 0.8 1.0 Probability of generated token 0 1 2 3 4 5 6 Density AQuA Standard CoT 0.0 0.2 0.4 0.6 0.8 1.0 Probability of generated token 0 5 10 15 20 Density GSM8K Standard CoT 0.0 0.2 0.4 0.6 0.8 1.0 Probability of generated…

**Figure 15.** Figure 15: Kernal density of Gemma2-27B [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

**Figure 16.** Figure 16: Results of Gemma2-2b. 0 10 20 30 40 50 Index of test samples 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Logits entropy COT Correct COT Wrong Standard Correct Standard Wrong (a) AQuA 0 10 20 30 40 50 Index of test samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Logits entropy COT Cor…

**Figure 17.** Figure 17: Results of Gemma2-9b. 0 10 20 30 40 50 Index of test samples 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Logits entropy COT Correct COT Wrong Standard Correct Standard Wrong (a) AQuA 0 10 20 30 40 50 Index of test samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Logits entropy COT Cor…

**Figure 18.** Figure 18: Results of LLaMA2-13b [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

**Figure 19.** Figure 19: Layer-wise activation differences across layers for models (3B, 8B, 70B) . [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[2021]

Kyle Palmieri was called for slashing

Are NLP models really able to solve simple math word problems? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics. Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy. 2024. Deciphering t...

arXiv 2022
[2024]

ROME: Memorization Insights from Text, Logits and Representation

A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Bo Li, Qinghua Zhao, and Lijie Wen. 2024. Rome: Memorization insights from text, logits and represen- tation.arXiv preprint arXiv:2403.00510. Dacheng Li, Shiyi Cao, Tyler Griggs,...

work page Pith review arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2021]

Kyle Palmieri was called for slashing

Are NLP models really able to solve simple math word problems? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics. Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy. 2024. Deciphering t...

arXiv 2022

[2] [2024]

ROME: Memorization Insights from Text, Logits and Representation

A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Bo Li, Qinghua Zhao, and Lijie Wen. 2024. Rome: Memorization insights from text, logits and represen- tation.arXiv preprint arXiv:2403.00510. Dacheng Li, Shiyi Cao, Tyler Griggs,...

work page Pith review arXiv 2024