REVIEW 4 major objections 6 minor 20 references
CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read CDW-CoT claims that per-instance prompt distributions built from cluster-specific optimized prompts consistently beat fixed CoT prompts across six reasoning benchmarks.
desk verdict Plausible incremental method, but the headline numbers are tuned on the test sets and the evaluation section has no variance estimates, so the central claim is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the distance-weighted prompt probability distribution. The candidate pool is initialized from the questions closest to each K-means cluster center, and each cluster $i$ is assigned an optimized probability vector $p^{(i)}$ over that pool, trained by black-box prompt learning. For a test instance, the effective distribution is $p = \sum_{i=1}^{K} w_i p^{(i)}$, with weights $w_i = \exp(-d_i/T) / \sum_j \exp(-d_j/T)$, where $d_i$ is the Euclidean distance to cluster center $i$ and $T$ is a temperature. This identity converts geometric proximity in embedding space into a per-instance blend of cluster-specific prompt strategies; the paper uses $T=0.3$ and a pool size of 40 for its main runs.
What would settle it
Swap the distance-weighted blend $p = \sum_i w_i p^{(i)}$ for a uniform blend of the same per-cluster distributions while holding clusters, pool, temperature, and sampling fixed; if uniform blending matches or beats distance weighting on the six benchmarks, the claim that proximity to cluster centers drives the improvement is falsified.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that a single global prompt distribution leaves accuracy on the table, while a prompt distribution optimized per cluster and then combined for each test instance by distance weighting does not. The paper reports exact-match accuracy gains on all six datasets for both LLaMA2-13B and LLaMA3-8B; the largest single jump is Letter, which rises from 30.80% to 82.67% on LLaMA2. The ablation results attribute part of the gain to clustering and part to distance weighting, with the full model beating both a nearest-cluster assignment and a no-clustering global distribution on every dataset.
Load-bearing premise
The load-bearing premise is that the ideal prompt for a question can be approximated by a weighted average of the prompt distributions of nearby clusters, with weights determined by embedding distance; if embedding distance does not track prompt usefulness, the reported gains may come from per-cluster optimization rather than from distance weighting.
Editorial extensions
If this is right
- Prompt count no longer has to scale with instance count: a fixed candidate pool plus per-cluster distributions serves an entire dataset.
- Per-cluster optimization can be done offline, and inference adds only an embedding lookup and a weighted average, so the same training effort supports any number of test instances.
- Datasets with heterogeneous subpopulations benefit most; the Letter jump from 30.80% to 82.67% on LLaMA2 suggests symbolic tasks are especially sensitive to prompt-instance matching.
- Temperature and pool size are material hyperparameters: the paper finds best accuracy at $T=0.3$ and pool size 40, so deployments should tune them rather than defaulting.
- Gains appear on both LLaMA2-13B and LLaMA3-8B, suggesting the mechanism is not tied to one model family.
Reading between the lines
- Not tested in the paper: comparing Eq. (7)'s distance-weighted blend against a uniform blend of the same per-cluster distributions would isolate whether proximity to cluster centers, rather than mixing itself, drives the gains.
- An implicit bet is that the sentence-transformer embedding space aligns with prompt usefulness; swapping the embedder and measuring accuracy drift would test whether the clusters, not the downstream model, carry the signal.
- Because cluster distributions are optimized independently, the method could be extended to streaming data by updating only the distributions of clusters that receive new instances.
- Boundary instances are where the blend is most sensitive to $T$; a testable extension is to set $T$ per instance from the entropy of the distance weights.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CDW-CoT, a chain-of-thought prompting method that clusters the training data, learns an optimized prompt probability distribution per cluster via a black-box policy-gradient update (referred to as BBPL), and at inference time constructs a per-instance prompt distribution as a temperature-scaled, distance-weighted combination of the cluster distributions. The method is evaluated on six reasoning datasets (CSQA, StrategyQA, Letter, Coin, MultiArith, AQuA) with LLaMA2-13B and LLaMA3-8B, reporting exact-match accuracies that claim consistent improvement over Zero-Shot-CoT, Auto-CoT, and Manual-CoT.
Significance. If the central empirical claim were properly supported, the idea of instance-adaptive prompt distributions would be a useful contribution to automatic chain-of-thought prompting: the ablation study (Table 3) gives some preliminary evidence that distance weighting and clustering each contribute to accuracy, and the prompt budget is kept small. However, the manuscript as written does not establish the headline claim of consistent out-of-sample improvements, because the key hyperparameters (temperature and pool size) appear to be selected on the same test sets used for the main results, and the reported accuracies come from a single run without statistical significance testing. The method description also contains internal inconsistencies in the optimization equations that prevent reproduction. The core idea is salvageable, but the current evidence is not sufficient.
major comments (4)
- [Sensitivity Analysis of Temperature / Impact of Pool Size on CDW-CoT / Table 1] The main results in Table 1 are not out-of-sample evaluations of a fixed configuration. In the section 'Sensitivity Analysis of Temperature', T=0.3 is selected using accuracy on StrategyQA and MultiArith, and in 'Impact of Pool Size on CDW-CoT', S=40 is selected on CommonsenseQA and MultiArith; these exact settings are then used for the headline results in Table 1. Concretely, Figure 4 reports MultiArith accuracy of 85.56 at S=40, which is identical to the Table 1 value, and Figure 3 selects T=0.3 on datasets that later appear in Table 1. This means the reported improvements for MultiArith, CSQA, and StrategyQA are at least partly tuned on the reporting test sets. The authors must either select hyperparameters on a held-out validation split that is disjoint from the final test set, or report results for configurations fixed before any test-set evaluation.
- [Experiments and Results / Table 1] All accuracy tables report a single run with no standard deviations, no number of seeds, and no significance tests. Some differences are very small, for example StrategyQA LLaMA3 (66.72 to 67.44, a 0.72-point increase) and CSQA LLaMA3 (69.57 to 72.15, a 2.58-point increase), yet the text describes them as consistent improvements. The central empirical claim requires either multiple random seeds with reported variance or a paired significance test (e.g., bootstrap over instances) to rule out noise. Please add these statistics or temper the claim accordingly.
- [Optimizing Prompt Probability Distributions for Clusters, Eqs. (1)-(5)] The optimization equations are internally inconsistent as written. Equation (1) defines δ = -1/p(i), a scalar quantity, but Eq. (2) then indexes it as δk,m,n and sets it to -δk,m,n if the prompt is selected and δk,m,n otherwise, without defining how the scalar δ becomes indexed or what the sign flip is supposed to accomplish. Equation (3) multiplies the loss deviation by δk, but δk has not been introduced. As written, this procedure cannot be implemented or checked. Please give a coherent derivation of the policy-gradient update, following the notation of the cited BBPL/BDPL references, or provide a precise algorithmic specification of the update.
- [Distance-Weighted Prompt Selection and Inference, Eq. (7)] The core assumption of the method—that a test instance's optimal prompt distribution is the convex combination of per-cluster prompt distributions weighted by a temperature-scaled softmax of Euclidean distances to cluster centers—is asserted without empirical or theoretical justification. The ablation in Table 3 shows that distance weighting outperforms nearest-cluster assignment on these datasets, which is supportive, but it does not validate the specific functional form of Eq. (7), particularly for instances near cluster boundaries where the interpolation is most sensitive to T. Please provide an analysis of the learned weights, a comparison to alternative interpolation schemes, or at least a validation-based sensitivity analysis with error bars.
minor comments (6)
- [Cluster-Based Prompt Candidate Pool Initialization] The pre-trained sentence transformer used for embeddings is never named; please identify the exact model so the experiments are reproducible.
- [Data Split and Number of Clusters Identification, Table 2] The number of clusters is manually adjusted per dataset (e.g., from the default 8 to 3 for Coin and MultiArith), but the criterion for adjustment is not stated. Please clarify whether these choices were made on a validation split and report their sensitivity.
- [Models and Baselines] The claim that CDW-CoT uses 'no more prompts than baselines' cannot be verified because the prompt counts for Zero-Shot-CoT, Auto-CoT, and Manual-CoT are not reported. Please provide the number of prompts used for each baseline.
- [Experiments and Results] Several baseline numbers are unusually low compared with published results for these models and datasets (e.g., Auto-CoT MultiArith 44.17 and Manual-CoT Letter 15.80 on LLaMA2-13B). Please clarify the evaluation settings or report these baselines with the same decoding and prompt-count conditions as the proposed method.
- [Sensitivity Analysis of Temperature / Impact of Pool Size on CDW-CoT] Figures 3 and 4 report accuracies without error bars or repeated runs, so it is difficult to assess whether the reported trends are stable; please add multiple runs and variance information.
- [Related Works and Algorithm 1] There is a typo in the description of Auto-CoT ('clustering similar questions and and using'), and the acronyms BDPL and BBPL are used inconsistently; please unify the terminology.
Circularity Check
Headline gains for CSQA, StrategyQA, and MultiArith are partially in-sample: temperature and pool size are selected on the test split whose accuracies are then reported in Table 1.
-
fitted input called prediction
[Sensitivity Analysis of Temperature; Impact of Pool Size on CDW-CoT; Prompt Engineering; Table 1]
"We varied the pool size from 10 to 40 ... For MultiArith, accuracy steadily improves from 73.33% at a pool size of 10 to 85.56% at a pool size of 40. Similarly, for CommonsenseQA, accuracy increases from 54.89% to 61.41% as the pool size grows. ... The model achieves optimal performance at a temperature of 0.3, striking an effective balance between specificity and sensitivity."
The S=40 values in the pool-size sweep are exactly the test-set accuracies in Table 1: for MultiArith, 85.56% is 77/90 on the 90-example test split, and for CSQA, 61.41% is 113/184 on the 184-example test split. These exact percentages cannot be produced from the evaluation splits listed in Table 2 (153 and 312 examples), so the sweep was run on the test instances. The temperature sweep likewise reports optimal T=0.3 on StrategyQA and MultiArith, and Table 1 uses T=0.3 with the same test values (70.06% and 85.56%). Consequently, temperature and pool size are selected by peeking at test labels, and the headline 'consistently outperforms' claim rests on configurations chosen to maximize accuracy on the very test split later reported as evidence.
full rationale
There is no definitional circularity, no imported uniqueness theorem, and no self-citation chain: the per-cluster prompt distributions are optimized on training data, and Eqs. (6)-(7) are openly stated design choices rather than derived consequences. The one substantive circularity-like defect is that the two key hyperparameters T and S are selected on the test split used for the headline table, as shown by the exact match between the Figure 3/4 values and Table 1 (77/90 on MultiArith, 113/184 on CSQA, and 70.06% on StrategyQA). Because the central claim is empirical, this makes the reported out-of-sample accuracy partially in-sample for at least three datasets; the ranking against baselines is therefore not established for a configuration fixed before test-set inspection. If the sensitivity sweeps had been run on the held-out evaluation split, the concern would disappear, but the paper does not say that, and the percentages do not fit the eval splits. Score 6 reflects one partial circularity affecting the central claim, not a fully forced derivation.
Assumptions & free parameters
free parameters (4)
- Number of clusters K per dataset =
7, 6, 4, 3, 3, 4 for CSQA, StrategyQA, Letter, Coin, MultiArith, AQuA
- Temperature T =
0.3
- Pool size S =
40
- Per-cluster prompt distributions p^(i) =
not reported
assumptions (4)
- domain assumption The optimal prompt distribution for a test instance is a convex combination of per-cluster optimized distributions weighted by Euclidean distance in SentenceTransformer embedding space (Eq. 7).
- domain assumption K-means clusters in SentenceTransformer embedding space correspond to groups with distinct optimal CoT prompts.
- domain assumption Zero-shot CoT applied to cluster centroids generates diverse and useful prompt candidates.
- ad hoc to paper The BBPL policy-gradient updates (Eqs. 1-5) yield valid and unbiased prompt probability updates.
Cite this review
Pith. "Pith review of CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning." pith.science (2026). https://pith.science/paper/ONDWJS2Q
@misc{pith2026250112226,
author = {Pith},
title = {Pith review of: CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ONDWJS2Q}},
note = {Machine review of arXiv:2501.12226}
}
read the original abstract
Large Language Models (LLMs) have recently achieved impressive results in complex reasoning tasks through Chain of Thought (CoT) prompting. However, most existing CoT methods rely on using the same prompts, whether manually designed or automatically generated, to handle the entire dataset. This one-size-fits-all approach may fail to meet the specific needs arising from the diversities within a single dataset. To solve this problem, we propose the Clustered Distance-Weighted Chain of Thought (CDW-CoT) method, which dynamically constructs prompts tailored to the characteristics of each data instance by integrating clustering and prompt optimization techniques. Our method employs clustering algorithms to categorize the dataset into distinct groups, from which a candidate pool of prompts is selected to reflect the inherent diversity within the dataset. For each cluster, CDW-CoT trains the optimal prompt probability distribution tailored to their specific characteristics. Finally, it dynamically constructs a unique prompt probability distribution for each test instance, based on its proximity to cluster centers, from which prompts are selected for reasoning. CDW-CoT consistently outperforms traditional CoT methods across six datasets, including commonsense, symbolic, and mathematical reasoning tasks. Specifically, when compared to manual CoT, CDW-CoT achieves an average accuracy improvement of 25.34% on LLaMA2 (13B) and 15.72% on LLaMA3 (8B).
Figures
Reference graph
Works this paper leans on
-
[2]
arXiv preprint arXiv:2311.04155
Black-box prompt optimiza- tion: Aligning large language models without model train- ing. arXiv preprint arXiv:2311.04155. Chu, Z.; Chen, J.; Chen, Q.; Yu, W.; He, T.; Wang, H.; Peng, W.; Liu, M.; Qin, B.; and Liu, T
-
[3]
arXiv preprint arXiv:2309.15402
A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402. Diao, S.; Huang, Z.; Xu, R.; Li, X.; Lin, Y .; Zhou, X.; and Zhang, T
-
[6]
arXiv preprint arXiv:2303.05398
Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y .; and Iwasawa, Y
-
[7]
arXiv preprint arXiv:2305.05181
Mot: Memory-of-thought enables chatgpt to self-improve. arXiv preprint arXiv:2305.05181. Li, Y .; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W
-
[8]
arXiv preprint arXiv:2206.02336
On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336. Li, Y .; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W
-
[10]
arXiv preprint arXiv:2209.14610
Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610. Lyu, Q.; Havaldar, S.; Stein, A.; Zhang, L.; Rao, D.; Wong, E.; Apidianaki, M.; and Callison-Burch, C
-
[11]
arXiv preprint arXiv:2301.13379
Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379. Roy, S.; and Roth, D
-
[13]
arXiv preprint arXiv:2302.12822
Automatic prompt augmentation and selection with chain-of-thought from la- beled data. arXiv preprint arXiv:2302.12822. Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J
Show all 20 references
-
[15]
arXiv preprint arXiv:2201.08239
Lamda: Language models for dialog appli- cations. arXiv preprint arXiv:2201.08239. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y .; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al
-
[16]
arXiv preprint arXiv:2307.09288
Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Wang, J.; Sun, Q.; Li, X.; and Gao, M
-
[17]
arXiv preprint arXiv:2306.06427
Boosting lan- guage models reasoning with chain-of-knowledge prompt- ing. arXiv preprint arXiv:2306.06427. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022a. Self-consistency improves chain of thought reasoning in language models...
-
[18]
arXiv preprint arXiv:2212.09561
Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561. Wu, D.; Zhang, J.; and Huang, X
-
[19]
arXiv preprint arXiv:2307.01640
Chain of thought prompting elicits knowledge augmentation. arXiv preprint arXiv:2307.01640. Zhang, Z.; Zhang, A.; Li, M.; and Smola, A
-
[20]
arXiv preprint arXiv:2210.03493
Auto- matic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493
-
[2016]
arXiv preprint arXiv:1608.01413
Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413. Shao, Z.; Gong, Y .; Shen, Y .; Huang, M.; Duan, N.; and Chen, W
-
[2017]
arXiv preprint arXiv:1705.04146
Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. Lu, P.; Qiu, L.; Chang, K.-W.; Wu, Y . N.; Zhu, S.-C.; Ra- jpurohit, T.; Clark, P.; and Kalyan, A
-
[2018]
arXiv preprint arXiv:1811.00937
Commonsenseqa: A question answering chal- lenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937. Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kul- shreshtha, A.; Cheng, H.-T.; Jin, A.; Bos, T.; Baker, L.; Du, Y .; et al
-
[2022]
arXiv preprint arXiv:2201.08531
Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al
-
[2023]
arXiv preprint arXiv:2305.11860
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs. arXiv preprint arXiv:2305.11860. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al
-
[2024]
arXiv preprint arXiv:2407.21783
The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Fu, Y .; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.