REVIEW 2 major objections 4 minor 20 references
Skills Know Their Neighbors: Cluster-Contrastive Capability Pages for Skill Retrieval
T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that a large share of skill-routing error is baked into skill documents, and that offline-compiled Capability Pages with positive and negative evidence improve first-stage recall and end-to-end task success without…
desk verdict A genuinely useful representation-side fix for skill retrieval, carefully evaluated, with one real gap (T- never validated against execution) and one missing control (a text-length-matched positive field in the router ablation). read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Capability Page, a per-skill text card compiled offline by comparing each skill with its nearest neighbors in embedding space. Each page carries three fields: a positive trigger $T^+$ (query forms that should select the skill), a negative boundary $T^-$ (confusable queries that belong to a named neighbor), and a discriminative body $B$ (the core formula or decision rule). The theoretical carrier is the document-imposed lower bound: when two documents are identical but executable regions differ, any scorer reading only $(q,x)$ must err on at least half the symmetric difference $\mathcal{R}(s_i) \triangle \mathcal{R}(s_j)$, so the only cure is changing the observation. The mechanism works by giving the retrieval index the positive view $(T^+, B, x)$ and giving the router the exclusion view $T^-$, which keeps negative descriptions out of similarity embeddings while still using them for comparative candidate selection.
What would settle it
Take a random sample of compiled $T^-$ statements, execute the focal skill on the queries its $T^-$ says belong to a neighbor, and measure how often the focal skill actually succeeds; a nontrivial failure rate would mean the boundary is unreliable and the reported +3.62 end-to-end gain would not generalize.
Extended reading notes
Core claim
The paper models a skill's capability as its executable region $\mathcal{R}(s_i)$—the set of queries the executor can actually solve—and treats the skill's document $x_i$ as a lossy observation of that region. From this it derives a diagnostic decomposition splitting retrieval error into document error (information absent from the text) and retriever error (failure to read what is there), plus a lower bound: if two skills have identical documents but different executable regions, every scoring function over the raw documents must make an error proportional to the functional difference between the skills. The proposed remedy is the Capability Page: a cluster-contrastive representation whose positive trigger $T^+$ and discriminative body $B$ expand what the retrieval index can match, while the negative boundary $T^-$ names neighboring skills' query regions and is reserved for the routing card, where it rejects confusable candidates. Empirically, the index view improves Recall@10 for BM25, TF-IDF, BGE-M3, and two Qwen3 embeddings, and the $T^-$ routing card improves end-to-end success over all four tested executors; the same fixed encoder reaches 73.07% MRR@50 on a Chinese skill library.
Load-bearing premise
The paper assumes the offline compiler's negative-boundary statements correctly mark queries the focal skill cannot solve, yet it verifies them only with a manual quality-control pass rather than by executing the skills.
Editorial extensions
If this is right
- Skill libraries can be improved text-side: rewriting the indexed skill documents yields recall gains for sparse and dense retrievers alike, with BM25 gaining 7.63 Recall@10 points.
- Keeping the negative boundary out of the similarity index and exposing it only on the routing card is the right division of labor; appending $T^-$ to the indexed tail hurts all three dense retrievers.
- The positive trigger and discriminative body contribute in different regimes: $T^+$ supplies query-form coverage, largest on LogicBench and ToolQA, while $B$ provides a compact capability fingerprint.
- The gains transfer across language and distribution: with a fixed encoder, $B$ adds 3.60 MRR@50 points and $T^+$ another 2.97 on a predominantly Chinese skill library.
- Capability Pages complement rather than replace model-side improvements, targeting the document term of the error decomposition while stronger encoders and rerankers target the retriever term.
Reading between the lines
- A natural next step is to audit generated negative boundaries by execution rather than manual review, feeding misclassified $T^-$ entries back into the compiler to close the boundary loop.
- The document-imposed lower bound suggests a diagnostic tool for any skill library: estimate document error versus retriever error by comparing an optimal reader on raw documents with the observed retriever, then decide whether to invest in better text or better models.
- The Capability Page schema could generalize beyond agent skills to API documentation, tool catalogs, or RAG corpus entries where near-duplicate text hides different behaviors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses skill retrieval for LLM agents, arguing that retrieval errors arise not only from the scorer but from the text being scored. It formalizes a skill's capability as its executable region R(s_i), the set of queries the skill can solve, and treats the skill document as a lossy observation of that region. The authors propose Capability Pages: offline-compiled, cluster-contrastive representations containing a positive trigger T+, a negative boundary T−, and a discriminative body B, appended to the original document. At inference time the retrieval index uses T+, B, and the original document, while the router receives T− as an explicit exclusion condition. On SRA-Bench (26,262 skills, 5,400 questions) the index view improves Recall@10 for all five tested retrievers, with a mean gain of 2.94 points; exposing T− on routing cards improves end-to-end task success by 3.62 points on average across four executors and six datasets. A transfer experiment on Chinese SSL-SkillDiscovery reaches 73.07% MRR@50 with a fixed encoder. The method requires no modification to online models, only offline rewriting of skill documents.
Significance. If the empirical claims hold, the work is practically significant: it shows that offline document rewriting can improve both first-stage retrieval and downstream routing without retraining or changing online models. The paper's strengths are its careful experimental design: leave-one-out field contributions, an index-with-T− ablation, a doc2query-style baseline, a top-1 preservation analysis, fixed-encoder transfer evaluation, and repeated-run uncertainty estimates with confidence intervals. The authors also honestly report cases where the representation does not help on every metric, such as the small nDCG@10 and MRR@10 decreases for Qwen3-Embedding-0.6B. The cross-executor consistency of the end-to-end gain is reassuring because the compiler and one executor are both DeepSeek-V4-Pro, yet the gains also appear for Qwen3.6 and Gemma executors. The main open risk is that the novel component, T−, is generated without execution-based validation of the boundary it describes, and the router ablation does not control for the added text length.
major comments (2)
- [§5.3, Table 5] The central end-to-end claim rests on an unverified premise: that the T− strings written by DeepSeek-V4-Pro correctly describe the boundary between the focal skill's executable region and its neighbors' regions. Eq. (6) defines T− as (∪_{j∈N_i} R(s_j)) \ R(s_i), but the compiler never observes execution outcomes and the manual quality-control pass judges clarity and distinctness, not whether the focal skill would actually fail on the queries described by T−. If a T− statement over-excludes, the router will reject the correct skill and task success will drop; the measured +3.62 average shows such errors do not dominate on SRA-Bench, but it does not validate boundary accuracy for transfer to new libraries or query distributions. I request an execution-based audit: sample generated T− statements, construct or extract queries of the forms they describe, run the focal skill and the named neighbor on those queries, and report how often the focal skill fails while the neighbor succeeds, separately for each executor or at least for the strongest and weakest executors.
- [§5.3, Table 5] The router ablation compares a base card containing the skill name and original description with a +T− card that additionally contains the T− field. Because the +T− arm always has strictly more text, the +3.62-point improvement cannot be attributed specifically to negative-boundary semantics; any additional discriminative text, or even a neutral filler of the same length, might help the router compare candidates. A control arm that appends an equal-length positive or neutral field (for example, the candidate's own T+ or B, or a benign paraphrase) is needed to isolate the contribution of the negative boundary. Without this control, the attribution of the end-to-end gain to T− as exclusion evidence is not cleanly established.
minor comments (4)
- [Appendix D] Appendix D refers to "main-paper Table 3" for the 95% confidence intervals, but the intervals appear in Table 5; the cross-reference should be corrected.
- [§5.2, §4.3] The sentence in §5.2 that T− "still cannot be produced by single-document expansion" is too strong because §4.3 describes a singleton prompt that infers likely confusable tasks from a single document alone. The statement should be qualified to cluster-contrastive T− or acknowledge the singleton case.
- [Table 8] The leave-one-out margins d(T+) and d(B) in Table 8 are defined in the table caption, but the main text refers to Figure 4 for the same quantities; it would help to state the definition consistently near the first mention of leave-one-out contributions.
- [§7] The limitations section addresses the recall-dependence of T− and the structural nature of the clustering sweep, but it does not mention the absence of execution-based validation of generated T− boundaries; adding that caveat would make the scope clearer.
Circularity Check
No significant circularity: the theoretical bound is self-contained and the headline gains are measured against external benchmarks rather than derived from the paper's own definitions.
full rationale
The paper's derivation chain consists of (1) a formal executable-region definition (Eqs. 1-2), (2) a document-imposed error decomposition and lower bound (Eqs. 3-5, proved in Appendix A), and (3) a design proposal for Capability Pages with fields T+, T-, and B. The lower bound is a self-contained mathematical statement that holds for any scorer reading identical documents; it does not assume the compiler output and is not used to compute the reported gains. The definition of T- in Eq. (6) is normative: it states what a negative boundary should encode, not that the LLM compiler automatically realizes that set. The headline claims—Recall@10 gains across five retrievers, the +3.62-point end-to-end improvement, and the SSL-SkillDiscovery MRR@50 transfer result—are all measured against external benchmarks with fixed candidate pools, fixed routing protocols, and no fitted parameters renamed as predictions. The self-citations to Wang et al. (2026b) are used only to distinguish this work from concurrent WRITE/SKIP retrieve-rerank training, and they are not load-bearing for any claimed result. The main weakness is that generated T- boundaries are checked only by manual quality control in Section 4.3 rather than against actual execution outcomes, and Section 7 notes that the routing benefit applies once the correct skill reaches the top k; this is an empirical generalizability concern, not circularity, because the +3.62 gain is measured rather than derived from an assumption that T- is correct.
Assumptions & free parameters
free parameters (2)
- KMeans cluster count k =
200
- Agglomerative cosine distance threshold =
0.15
assumptions (5)
- domain assumption Each skill has a well-defined executable region R(s_i) with an objective success test.
- domain assumption Embedding-space cosine similarity groups confusable skills into clusters that correspond to meaningful executable-region boundaries.
- ad hoc to paper The generated T− text correctly reflects the true boundary between executable regions.
- domain assumption The router can correctly interpret T− exclusion statements, including cross-references to sibling skill_ids, and use them to reject confusable candidates without over-rejecting.
- ad hoc to paper The document-imposed lower bound (Eq. 5) extends conceptually from exactly identical documents to the near-identical documents found in practice.
Cite this review
Pith. "Pith review of Skills Know Their Neighbors: Cluster-Contrastive Capability Pages for Skill Retrieval." pith.science (2026). https://pith.science/paper/O7HZKB2R
@misc{pith2026260804482,
author = {Pith},
title = {Pith review of: Skills Know Their Neighbors: Cluster-Contrastive Capability Pages for Skill Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/O7HZKB2R}},
note = {Machine review of arXiv:2608.04482}
}
abstract
As skill libraries grow, large language model agents must retrieve reusable skills from candidates that often share the same topic and vocabulary but implement different capabilities. Retrieval is limited not only by the scorer but also by the text being scored: a document may describe what a skill does without stating which similar requests should be routed elsewhere. We formalize a skill's capability as its \emph{executable region}, the set of queries it can solve, and view its document as a lossy observation of that region. This view exposes a document-imposed component of retrieval error that cannot be removed by improving the retriever alone. We therefore propose \emph{Capability Pages}, cluster-contrastive skill representations containing a positive trigger $\Tpos$, a negative boundary $\Tneg$, and a discriminative body $B$. An offline compiler compares neighboring skills to write these fields. At inference time, the index uses $\Tpos$ and $B$ for candidate recall, while the router uses $\Tneg$ to reject confusable alternatives. On SRA-Bench, which contains 26{,}262 skills and 5{,}400 questions from six datasets, Capability Pages improve Recall@10 for all five tested retrievers, with a mean gain of $2.94$ points. Adding $\Tneg$ to candidate cards improves end-to-end task success by $3.62$ points on average across four executors and six datasets. A transfer evaluation on Chinese SSL-SkillDiscovery reaches $73.07\%$ MRR@50 using the same encoder across conditions. Capability Pages require no modification to the online models; they improve routing by rewriting the offline skill library.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
SkillDAG: Self-evolving typed skill graphs for LLM skill selection at scale
Tong Bai, Zhenglin Wan, Xingrui Yu, and Pengfei Zhou. SkillDAG: Self-evolving typed skill graphs for LLM skill selection at scale. arXiv preprint arXiv:2606.03056,
-
[4]
DeepSeek-V4: Towards highly efficient million-token context intelligence
DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348,
-
[5]
SkillResolve-Bench: Measuring and resolving same-capability ambiguity in agent skill retrieval
Jiandong Ding. SkillResolve-Bench: Measuring and resolving same-capability ambiguity in agent skill retrieval. arXiv preprint arXiv:2606.10388,
-
[6]
Precise zero-shot dense retrieval without rele- vance labels
Luyu Gao, Xinyu Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without rele- vance labels. arXiv preprint arXiv:2212.10496,
-
[9]
Qiliang Liang, Hansi Wang, Zhong Liang, and Y ang Liu. From skill text to skill structure: The scheduling-structural-logical representation for agent skills. arXiv preprint arXiv:2604.24026 ,
-
[10]
Graph-of-skills: Dependency-aware structural retrieval for massive agent skills
Dawei Liu, Zongxia Li, Hongyang Du, Xiyang Wu, Shihang Gui, Y ongbei Kuang, and Lichao Sun. Graph-of-skills: Dependency-aware structural retrieval for massive agent skills. arXiv preprint arXiv:2604.05333,
-
[11]
Trace2Skill: Distill trajectory-local lessons into transfer- able agent skills
Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transfer- able agent skills. arXiv preprint arXiv:2603.25158,
-
[12]
Document expansion by query prediction
10 Rodrigo Nogueira, Wei Y ang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375,
arXiv 1904
Show all 20 references
-
[14]
Yiqun Sun, Pengfei Wei, and Lawrence B. Hsieh. Don’t retrieve, navigate: Distilling enterprise knowledge into navigable agent skills for QA and RAG. arXiv preprint arXiv:2604.14572,
-
[15]
Gemma 4 technical report
Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770,
-
[16]
SkillX: Automatically constructing skill knowledge bases for agents
Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Y ao, Runnan Fang, Shuofei Qiao, Kexin Cao, Guozhou Zheng, Xiang Qi, Peng Zhang, and Shumin Deng. SkillX: Automatically constructing skill knowledge bases for agents. arXiv preprint arXiv:2604.04804, 2026a. Zifei Wang, Wei Wen, Qiang...
-
[17]
SkillSmith: Compiling agent skills into boundary-guided runtime interfaces
Duling Xu, Zheng Chen, Zaifeng Pan, Jiawei Guan, Dong Dong, Jialin Li, and Bangzheng Pu. SkillSmith: Compiling agent skills into boundary-guided runtime interfaces. arXiv preprint arXiv:2605.15215,
-
[18]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Y ang, Anfeng Li, Baosong Y ang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[19]
Qwen3 embedding: Advancing text embedding and reranking through foundation models
Yulong Zhang, Minghui Li, Dayiheng Long, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176,
-
[20]
SkillRouter: Skill routing for LLM agents at scale
Y anZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuai Zhu, Y ong Wu, Tianze Xu, Bao- hua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu. SkillRouter: Skill routing for LLM agents at scale. arXiv preprint arXiv:2603.22455,
-
[2009]
Skill retrieval augmentation for agentic AI
Weihang Su, Jianming Long, Qingyao Ai, Yichen Tang, Changyue Wang, Yiteng Tu, and Yiqun Liu. Skill retrieval augmentation for agentic AI. arXiv preprint arXiv:2604.24594,
-
[2023]
SkillWiki: A living knowledge infrastruc- ture for agent skills
Dingcheng Huang, Yuda Ding, Bingshuo Liu, Qingbin Liu, Xi Chen, Jiang Bian, Hongliang Sun, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, and Dianbo Sui. SkillWiki: A living knowledge infrastruc- ture for agent skills. arXiv preprint arXiv:2606.16523, 2026a. Shouzheng Huang, Meishan Zha...
-
[2024]
SkillRet: A large-scale benchmark for skill retrieval in LLM agents
Hongcheol Cho, Ryangkyung Kang, and Y oungeun Kim. SkillRet: A large-scale benchmark for skill retrieval in LLM agents. arXiv preprint arXiv:2605.05726,
-
[2025]
SkillsBench: Benchmarking how well agent skills work across diverse tasks
Xiangyi Li, Wenbo Chen, Yimin Liu, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670,
-
[2026]
M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024 , pp. ...
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.