Chunk-Distilled Language Modeling

Jiawei Zhou; Karen Livescu; Yanhong Li

REVIEW 3 major objections 7 minor 17 references

Chunk-Distilled Language Modeling

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CD-LM claims that chunk-level retrieval from a trie datastore can speed up decoding, improve perplexity, and inject new knowledge into any off-the-shelf language model, all without training.

desk verdict The exact sequence-probability DP is a genuine contribution; the efficiency metrics need a serious caveat before trusting the speedups. read the letter →

arxiv 2501.00343 v1 pith:ZAIAU7Y2 submitted 2024-12-31 cs.CL cs.AI

Yanhong Li , Karen Livescu , Jiawei Zhou This is my paper · ORCID

classification cs.CLcs.AI

keywords chunk-distilledlanguagemodelingretrieval-augmentedgenerationtraining-freeknowledgedistillationspeculativedecodingtriedatastoreperplexitydynamicprograminferenceefficiencydomainadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chunk-Distilled Language Modeling (CD-LM) is a training-free way to combine an off-the-shelf autoregressive language model with a retrieval module that proposes multi-token chunks. At each decoding step the model either emits a single token or accepts a retrieved chunk as the continuation, skipping several autoregressive forward passes at once. The paper derives an exact dynamic program, similar to a backward algorithm, for the probability of a sequence under this interleaved process, so perplexity measures real CD-LM behavior. With chunks drawn from a stronger teacher, from the model itself, or from expert-curated corpora, CD-LM is reported to improve perplexity and domain fit, speed up decoding, and inject facts or private information without any parameter updates.

What carries the argument

The load-bearing object is a trie-structured chunk datastore. Each stored chunk is a path in a trie rooted at its entry token (the last token before the chunk), and each node stores hidden-state vectors of the preceding contexts that lead to that chunk, so chunks of variable length can be retrieved together with their context. Retrieval takes the LM's current context vector, restricts the search to the trie of the last generated token via cosine similarity, and proposes the best-matching chunk with an acceptance probability obtained from a piecewise-linear map. The probability model uses latent Bernoulli variables for chunk acceptance and two backward recursions, $\alpha_n$ and $\beta_n$, to marginalize over them, which is what makes the reported perplexities exact for the CD-LM generative process.

What would settle it

Run the SCD-LM MT-Bench-80 setup on the same hardware while timing the full inference loop, including trie lookup and cosine similarity; if total tokens per second does not improve over the base LM at the same quality threshold, the efficiency claim fails. A second check is to sample many sequences from the CD-LM process and compare empirical frequencies with the dynamic-program probabilities: any systematic mismatch would show the perplexity formula misses a dependency.

Watch

Extended reading notes

Core claim

CD-LM's central claim is that chunk-level retrieval can improve language modeling performance and inference efficiency at the same time, without training. The generative process introduces a latent binary variable at each position that decides whether a retrieved chunk is accepted or the base LM's next token is used, and the contribution is a backward-style dynamic program (paired quantities $\alpha_n$ and $\beta_n$) that marginalizes over those decisions and yields exact sequence probabilities, hence perplexity, under CD-LM. The same mechanism is instantiated in three modes: chunks extracted from a larger or specialized teacher (KCD-LM), chunks extracted from the base model's own high-probability spans (SCD-LM), and chunks supplied by human or expert sources (ECD-LM). The paper reports that KCD-LM lowers GPT-2-small's perplexity on WikiText, medical, law, and code data relative to both the base LM and kNN-LM, SCD-LM saves a large fraction of forward passes on MT-Bench, and ECD-LM increases entity diversity and PII accuracy, all without updating parameters.

Load-bearing premise

The load-bearing assumption is that retrieval overhead can be ignored when measuring speed: the reported gains count saved forward passes and per-token decode time, but not the trie search and cosine-similarity computation that must run at every step, and the paper itself notes that retrieval is not optimized.

Editorial extensions

If this is right

KCD-LM provides a training-free distillation route: a small base model's perplexity can be pulled toward a larger teacher's, and on the tested domains the reported perplexity matches or beats directly fine-tuning the base model.
SCD-LM turns a model's own high-probability chunks into a reusable cache; repeated or paraphrased queries decode fewer tokens, saving forward passes and token time, and the savings grow when each question gets its own tailored datastore.
ECD-LM makes external knowledge available at inference time: the base model emits rare facts it otherwise would not produce, and private-information retrieval accuracy on a 1.5B model rises from 0% to 75.7% in the paper's PII experiment.
Because sequence probabilities are computed exactly under the CD-LM distribution, perplexity can serve as an intrinsic objective for guiding datastore construction and retrieval thresholds, not just a reporting metric.
Chunk storage reduces datastore size to roughly 30–40% of a token-level kNN datastore, and entry-token tries cut each search to about 0.0003–0.01% of the full store.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If retrieval latency were included in the timing, the reported efficiency gains would shrink; a fair system comparison should report end-to-end wall-clock throughput, not only forward passes saved.
The differentiable dynamic program opens a route the paper leaves implicit: train the acceptance mapping, the similarity metric, or even the base LM against the CD-LM likelihood, turning chunk retrieval into a learnable latent-variable model.
ECD-LM's PII experiments suggest personalized, private datastores as a training-free alternative to fine-tuning; the same design would need explicit access-control guarantees, since retrieval makes memorized private strings directly retrievable.
CD-LM's hard chunk acceptance is the sparse, multi-token analogue of kNN-LM's soft interpolation, so it should inherit kNN-LM's sensitivity to datastore quality and domain mismatch: one testable prediction is that gains shrink when the datastore is noisy or out-of-domain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the latent-variable formulation with the backward dynamic program for exact sequence probabilities is the real deal. I checked the recurrence in Section 5 and Appendix A: it correctly marginalizes over chunk acceptance for a deterministic retrieval proposal, and it is not present in the cited prior work. That is a solid theoretical contribution. Second, the efficiency claim in the abstract is load-bearing, but the metrics as reported almost certainly overstate wall-clock gains. TTS and FPS do not clearly include the cost of running the base LM over accepted chunk tokens to refresh the KV cache, nor the retrieval similarity search. The paper itself says retrieval is not optimized, and Appendix F.8 shows dozens of retrievals per generation. So treat the speedups with caution.

What the paper does well: it gives a clean, training-free way to inject chunks from virtually any source into an off-the-shelf LM. The experimental coverage is broad—three settings, four datasets for KCD, three models for SCD, and two ECD applications including private data injection. The PPL numbers under the CD-LM distribution are consistent and beat kNN-LM at comparable datastore sizes. That is credible.

Soft spots, in order of severity. First, the efficiency metrics need an explicit statement of what is included. FPS counts forward passes, but a forward pass over an accepted chunk is not free when you need the next retrieval query; if the implementation reruns the base LM over the chunk, the savings shrink substantially. Second, there are no error bars or significance tests anywhere in the main tables; all comparisons appear to be single runs. Third, the "distillation" framing is generous—this is retrieval with a hard acceptance decision—but the paper does compare against kNN-LM and RETOMATON, so the empirical claim stands on its own. Fourth, the abstract promises code and data, but the current version makes neither available; the math is reproducible by hand, but the retrieval pipeline details would matter for replication.

Who this is for: anyone working on efficient decoding or retrieval-augmented generation. The dynamic program is worth citing for the probabilistic treatment alone. The paper deserves a full peer review; the efficiency section needs a rewrite and a controlled wall-clock benchmark, but the core idea and the mathematics are solid.

Referee Report

3 major / 7 minor

Summary. The paper proposes Chunk-Distilled Language Modeling (CD-LM), a training-free inference-time method that interleaves standard autoregressive token generation with retrieval of multi-token chunks from a trie-structured datastore. Three variants are introduced: KCD-LM (chunks distilled from a teacher LM), SCD-LM (self-distilled chunks for speed), and ECD-LM (expert-curated chunks). The central theoretical contribution is a latent-variable formulation of the generative process and a dynamic program (Section 5, Appendix A) that computes exact sequence probabilities, enabling perplexity evaluation under the CD-LM distribution. Empirically, the paper reports PPL improvements for KCD-LM over a GPT-2 small base across four domains, speedups for SCD-LM on MT-Bench, and factual entity injection for ECD-LM. The probabilistic derivation appears mathematically sound, but the efficiency evaluation has important gaps: the reported FPS/TTS metrics do not account for retrieval and KV-cache refresh costs, and the comparison between KCD-LM and kNN-LM at matched datastore sizes is not fully specified.

Significance. If the dynamic program and the empirical claims hold, CD-LM offers a valuable way to measure and improve language modeling under chunk-based decoding, and the exact marginalization over chunk acceptance is a nice contribution. The method is training-free, which is attractive for domain adaptation without gradient updates. The paper ships detailed appendix material, including derivations and ablations. However, the load-bearing efficiency claim is currently supported by metrics that may overstate the wall-clock benefits, and the kNN-LM comparison is not reproducible as written. The PPL results are interesting but should be interpreted carefully because the model explicitly allows copying from an in-domain datastore.

major comments (3)

[Section 6.2, Tables 5-6, Eq. (3)] The efficiency metrics TTS and FPS are not defined precisely enough to support the paper's efficiency claims. FPS counts forward passes saved, but a forward pass over a chunk of length T after its acceptance is required to refresh the KV cache and obtain the hidden state used for the next retrieval query in Eq. (3); the FLOPs of that batched pass scale with T, not with one token. Thus the reported 43.33% FPS reduction for GPT-2-XL (Table 5) overstates compute savings. Similarly, TTS is reported as 'decoding time per token' without stating whether retrieval (cosine-similarity search over trie nodes) and the catch-up forward pass are included. The paper's conclusion explicitly says retrieval is not optimized, and Appendix F.8 shows tens of retrievals per generation, so if TTS excludes these costs, the end-to-end speedup may be substantially smaller. Please report wall-clock time per token with all components itemized, or clearly state what is excluded and soften the efficiency claim accordingly.
[Appendix E.3, Figure 10, Table 11] The claim that KCD-LM outperforms kNN-LM 'under the same datastore sizes' is not reproducible because the construction of the kNN-LM datastore at each size is not described. kNN-LM typically stores one key-value pair per token, whereas CD-LM stores chunks, so it is unclear how a kNN-LM datastore of, say, 46 GB was created. Is the chunk datastore used directly as a kNN-LM datastore (by tokenizing chunks), or are keys subsampled from the full corpus? The role of the chunk extraction threshold γ in the kNN-LM rows of Table 11 is also ambiguous. Please specify the exact procedure so the comparison can be reproduced and assessed.
[Section 6.2, Table 17, Figure 6] The observation that SCD-LM-generated text has lower perplexity under the base LM than the base LM's own generations is interpreted as 'the quality of generation benefits from explicit self-memories.' Perplexity of generated text under a fixed LM is not a valid quality metric; it measures how predictable the text is to that LM, and a decrease can simply reflect that the model is emitting repeated high-probability chunks from the datastore, i.e., reduced diversity. To support the quality claim, please report additional metrics such as n-gram diversity/repetition rate, or rely on the human/LLM-judge evaluations with the full breakdowns. If the lower PPL is due to repetition, the interpretation should be revised.

minor comments (7)

[Section 4.3] The sentence 'the chunk datastore construction this way only needs one forward pass of MθT on C' is misleading because a forward pass of the base model Mθ over the chunk contexts is also required to build the context-vector keys.
[Throughout] Typos include 'knolwedge' (Section 1), 'senario' (Section 6.3.2), 'publically' (Ethics Statement), and 'use to measure' (Section 5).
[Table 9, Section 6.3.2] Table 9 lists only GPT2-XL and LLaMA-2 despite the setup text stating that Mistral-7B-Instruct-v0.2 was also evaluated; please add the missing row or clarify.
[Section 6.2] The main text would benefit from an explicit statement about whether TTS includes retrieval and cache-refresh time; currently the reader must infer it from the appendix.
[Figure 5/10] Clarify whether the 'Datastore Size (GB)' axis refers to the CD-LM datastore only, and how kNN-LM is evaluated at the same size.
[Eq. (4)] Equation (4) uses $1\{n; z_{1:n}\}$ while the text below it mentions $1\{n; z_{2:n}\}$; please harmonize the notation.
[Table 2] The MAUVE scores are near zero for all models; although the footnote explains that relative comparisons are meaningful, the absolute values are hard to interpret and the reader should be pointed to the scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CD-LM's PPL dynamic program is a genuine marginalization of its own generative process, and the efficiency and quality claims are empirical rather than definitionally forced.

full rationale

The paper's central derivation is the Section 5 dynamic program for sequence probabilities, which marginalizes the latent chunk-acceptance variables z_n under the generative process defined in Section 3.2. The recursions in Eq. (5) and the final marginal in Eq. (6) follow from the model definition rather than from any fitted quantity renamed as a prediction. The hyperparameters gamma and eta are tuned on validation sets and then reported on test sets, which is standard practice; the KCD-LM perplexity numbers are evaluated on held-out test sequences with a datastore built from training corpora, and the kNN-LM and RETOMATON baselines use the same datastore. The ECD-LM experiments use externally curated Wikipedia hyperlink chunks and measure entity coverage and factual accuracy, not quantities that are defined by the chunk extraction itself. The SCD-LM testbeds build datastores from prior responses to the same or similar prompts, which is an explicitly stated repeated-query use case, and the REST baseline is evaluated on the same datastore, making the comparison controlled. No load-bearing step relies on a self-citation or an imported uniqueness theorem; related work such as REST and kNN-LM is cited and compared empirically. Concerns about wall-clock retrieval overhead or about the choice of testbed for SCD-LM are correctness or evaluation-validity issues, not circular derivation, so the paper's claimed derivation chain is self-contained.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or ontological entities; it is a probabilistic inference procedure over existing tokens. The free parameters are the thresholds gamma and eta, tuned on validation data. The core assumptions are about the reliability of vector-space context matching and the usefulness of teacher probability-based chunk extraction.

free parameters (2)

chunk extraction threshold gamma = 0.3 to 0.9, selected per dataset (0.4 for WikiText and Law, 0.3 for Code and Medical)
Tokens above this probability are grouped into chunks. Chosen per dataset to minimize validation PPL, so it is a free parameter fitted to the data.
retrieval acceptance threshold eta = 0.9995 for KCD-LM; 0.8 for GPT-2, 0.7 for LLaMA-2 and Mistral in SCD-LM
Minimum cosine similarity for a chunk to be accepted. Selected by tuning on validation sets using PPL and generation quality metrics.

assumptions (3)

domain assumption Cosine similarity in the LM's hidden space reliably identifies contextually appropriate chunk continuations
The retrieval model G assumes that the stored context vector nearest to the current context vector yields a valid continuation. This is assumed throughout Section 4.2.
domain assumption Teacher LM token probabilities above gamma identify reusable chunks
Chunk extraction (Section 4.3) relies on a thresholding heuristic where tokens with probability at least gamma form chunks. The paper provides examples but no proof that this identifies semantically coherent units.
standard math Standard probability calculus for latent variable marginalization
The dynamic program in Section 5 and Appendix A uses standard rules of conditional probability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Chunk-Distilled Language Modeling." pith.science (2026). https://pith.science/paper/ZAIAU7Y2

@misc{pith2026250100343,
  author       = {Pith},
  title        = {Pith review of: Chunk-Distilled Language Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAIAU7Y2}},
  note         = {Machine review of arXiv:2501.00343}
}

read the original abstract

We introduce Chunk-Distilled Language Modeling (CD-LM), an approach to text generation that addresses two challenges in current large language models (LLMs): the inefficiency of token-level generation, and the difficulty of adapting to new data and knowledge. Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step. Our retrieval framework enables flexible construction of model- or domain-specific datastores, either leveraging the internal knowledge of existing models, or incorporating expert insights from human-annotated corpora. This adaptability allows for enhanced control over the language model's distribution without necessitating additional training. We present the CD-LM formulation along with performance metrics demonstrating its ability to improve language model performance and efficiency across a diverse set of downstream tasks. Code and data will be made publicly available.

Figures

Figures reproduced from arXiv: 2501.00343 by the authors.

**Figure 2.** LLM token probabilities for the sentence: “ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of CD-LM. Colored text spans are generated together by chunk retrieval, inter [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: A graphical model illustration of the probabilistic model of CD-LM. The token sequence xn nodes are observed, and chunk acceptance variables zn are latent, governing how many tokens are to be generated at one step. Formally, we use n to index sequential token positio…

**Figure 5.** Figure 5: Comparison between KCD-LM and kNN-LM on PPL, along with datastore sizes con [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: SCD-LM performance on MT-Bench-80 with varying retrieval similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 8.** Figure 8: Additionally, the LLM-as-a-judge results presented in Appendix G.8 confirm that ECD [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

**Figure 7.** Figure 7: Distribution plot for GPT2-xlconversational when answering knowledgeintensive questions about Alan Turing. The plot compares the frequency versus rank of entities in generated responses for the Base LM vs. CD-LM. Similar trends were observed for LLaMA-2-7b-chat and …

**Figure 9.** Figure 9: Piecewise function mapping for qn. For each dataset and chunk extraction token probability threshold γ, we experiment gϕ with the above parametrization. We find that the second type of mapping function is a simple and effective approach. Therefore, we report the result…

**Figure 10.** Figure 10: Comparison between KCD-LM and kNN-LM on PPL, along with datastore sizes con [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: SCD-LM efficiency and generation performance on MT-Bench-10 with varying retrieval [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]

**Figure 12.** Figure 12: Distribution plot for GPT2-xl-conversational, LLaMA-2-7b-chat, and Mistral-7b models [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]

**Figure 13.** Figure 13: Screenshots of the human evaluation questionnaire. [PITH_FULL_IMAGE:figures/full_fig_p036_13.png]

**Figure 14.** Figure 14: Screenshots of the human evaluation questionnaire. [PITH_FULL_IMAGE:figures/full_fig_p037_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[1]

The PPLs can also guide the con- struction of CD-LM such as the datastore and retrieval modeling variations, to better fit the data of interest

[α2q2 + β2(1 − q2)] Indeed, any predictive probabilities can be computed as p(x∗ n:N |x∗ <n) =αnqn + βn(1 − qn) With this we can compute the perplexity (PPL) of any given text sequence under CD-LM, providing an intrinsic measure of our language modeling performance. The PPLs can also guide the con- struction of CD-LM such as the datastore and retrieval mo...

work page
[2]

Alm → Bret → Cret → Dlm → Elm

work page
[3]

20 For simplicity, we assume each token probability of LM is 0.3: Plm(A) =Plm(B|A) =

Alm → Blm → Cret → Dret → Elm 16Batch computation for multiple sequences may still be challenging as the proposed chunk lengths may not be aligned. 20 For simplicity, we assume each token probability of LM is 0.3: Plm(A) =Plm(B|A) = ... = Plm(E|A, B, C, D) = 0.3. Then we assume the probability of accepting a chunk is 0.5: qc1 = qc2 = 0.5. For Alm → Blm → ...

work page 2020
[8]

Alm → Blm → Clm → Dlm → Elm

work page
[11]

Chunk Parsing - We parse the corpus C into 512-token chunks with a stride of 448 tokens

work page
[12]

This is because the first 64 tokens of a chunk do not have enough preceding text to provide adequate context

Context Identification - To ensure each saved chunk has sufficient context, we disregard the first 64 tokens of each chunk during datastore construction. This is because the first 64 tokens of a chunk do not have enough preceding text to provide adequate context. - For a chunk starting at position i in the corpus, we consider the context to be the text fr...

work page 1948
[13]

The parametrization ϕ is none

Identity function: Here q = gϕ(s∗) = s∗. The parametrization ϕ is none. This mapping function only works when s∗ ≥ 0, which is mostly observed

work page
[14]

Iron Man

Piecewise linear function: We define a starting similarity score η ≥ 0 as the point corre- sponding to q = 0, and then the similarity score range of [η, 1] linearly maps to [0, 1] in the probability space of q. Specifically, q = gϕ(s∗) = ( 0 if s∗ < η, s∗−η 1−η if s∗ ≥ η. The mapping function is also illustrated in Figure 9. Here the parametrization ϕ inc...

work page

Show all 17 references

[15]

This theoretical model provided mathematicians and philosophers with a framework for study- ing the fundamental limits of computation

The Turing Machine (TM): Turing introduced the concept of the Turing machine, an abstract hypothetical machine that manipulates symbols on a strip of tape according to a set of rules. This theoretical model provided mathematicians and philosophers with a framework for study- i...
[16]

His ideas on computation and algorithms have laid the foundations for our understanding of these concepts

The Church ECD- LM Alan Turing, an English mathematician, logician, cryptanalyst, and theoretical biologist, is widely considered to be the father of theoretical computer science and artificial intelligence. His ideas on computation and algorithms have laid the foundations for...
[17]

It showed that all computable functions can be computed by a single abstract machine when configured with specific tape symbols and rules

Turing machine: This abstract machine introduced in 1936 is a model of computation that effectively defines an abstract model of a general-purpose computer. It showed that all computable functions can be computed by a single abstract machine when configured with specific tape ...

1936
[18]

Code Contributor

Church–Turing thesis: The Church-Turing thesis posits that every effectively calculable Prefix <—USER—> How did Turing’s personal life impact his career and legacy? <—ASSISTANT—> Base LM Alan Turing’s personal life, particularly his homosexuality, had a significant impact on h...

1951
[218]

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu

URL https://aclanthology.org/2022.findings-emnlp.218. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the As...

2022
[704]

15 Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag

URL https://aclanthology.org/2020.acl-main.704. 15 Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models, 2024. URL https://arxiv.org/ abs/2403.03870. Weijia Shi, Sewon Min, Michihiro Yasuna...

2020 arXiv
[2020]

Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen tau Yih, and Xi Victoria Lin

URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/6b493230205f780e1bc26945df7481e5-Paper.pdf. Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen tau Yih, and Xi Victoria Lin. Nearest neighbor speculative decoding for llm generation and attribution, ...

2020 arXiv
[2021]

URL https://aclanthology.org/2021.tacl-1

doi: 10.1162/tacl a 00371. URL https://aclanthology.org/2021.tacl-1. 22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Sto- ica. Judging LLM-as-a-Judge wit...

2021 doi
[2024]

doi: 10.18653/v1/2024.naacl-long.365

Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.365. URL https://aclanthology.org/2024.naacl-long.365. Xiang Gao, Michel Galley, and Bill Dolan. MixingBoard: A knowledgeable stylized integrated text generation platform. In Asli Celikyilmaz and Tsung...

2024 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

The PPLs can also guide the con- struction of CD-LM such as the datastore and retrieval modeling variations, to better fit the data of interest

[α2q2 + β2(1 − q2)] Indeed, any predictive probabilities can be computed as p(x∗ n:N |x∗ <n) =αnqn + βn(1 − qn) With this we can compute the perplexity (PPL) of any given text sequence under CD-LM, providing an intrinsic measure of our language modeling performance. The PPLs can also guide the con- struction of CD-LM such as the datastore and retrieval mo...

work page

[2] [2]

Alm → Bret → Cret → Dlm → Elm

work page

[3] [3]

20 For simplicity, we assume each token probability of LM is 0.3: Plm(A) =Plm(B|A) =

Alm → Blm → Cret → Dret → Elm 16Batch computation for multiple sequences may still be challenging as the proposed chunk lengths may not be aligned. 20 For simplicity, we assume each token probability of LM is 0.3: Plm(A) =Plm(B|A) = ... = Plm(E|A, B, C, D) = 0.3. Then we assume the probability of accepting a chunk is 0.5: qc1 = qc2 = 0.5. For Alm → Blm → ...

work page 2020

[4] [8]

Alm → Blm → Clm → Dlm → Elm

work page

[5] [11]

Chunk Parsing - We parse the corpus C into 512-token chunks with a stride of 448 tokens

work page

[6] [12]

This is because the first 64 tokens of a chunk do not have enough preceding text to provide adequate context

Context Identification - To ensure each saved chunk has sufficient context, we disregard the first 64 tokens of each chunk during datastore construction. This is because the first 64 tokens of a chunk do not have enough preceding text to provide adequate context. - For a chunk starting at position i in the corpus, we consider the context to be the text fr...

work page 1948

[7] [13]

The parametrization ϕ is none

Identity function: Here q = gϕ(s∗) = s∗. The parametrization ϕ is none. This mapping function only works when s∗ ≥ 0, which is mostly observed

work page

[8] [14]

Iron Man

Piecewise linear function: We define a starting similarity score η ≥ 0 as the point corre- sponding to q = 0, and then the similarity score range of [η, 1] linearly maps to [0, 1] in the probability space of q. Specifically, q = gϕ(s∗) = ( 0 if s∗ < η, s∗−η 1−η if s∗ ≥ η. The mapping function is also illustrated in Figure 9. Here the parametrization ϕ inc...

work page