{"id":"6e827a74-d673-4428-bb9d-739630c07a1c","arxiv_id":"2606.03565","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Retaining LLM SKIP rejections as query-conditional compatibility supervision, and using them in a cross-encoder reranker rather than a bi-encoder, improves agent skill routing on a new bilingual benchmark.","lead":"Skill retrieval for LLM agents is not document retrieval: top-K success also depends on whether skills can work together under a query. The authors keep LLM rejection decisions as compatibility labels, release a bilingual skill-routing benchmark, and train a two-stage retriever that improves joint set retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing concern is whether LLM WRITE/SKIP joint-plausibility is a faithful enough oracle for true query-conditioned skill compatibility C(q,S*_q).","rationale":"The reader correctly isolates the LLM-oracle assumption as the weakest link under the strongest claim. The rest of the paper is internally coherent: the relevance/compatibility factorization is clear, the bilateral-balancing gradient argument (Appendix B / Theorem 1) explains why SKIP is weak in the tested bi-encoder InfoNCE and useful as graded cross-encoder labels, and the Table 10 ablation matches that split. Gains on SkillRet and on Set-Compat are real under the paper’s own labels. Because the concern is already acknowledged and does not invent an inconsistency, the appropriate adjustment is none: keep CONDITIONAL, with the same medium correctness risk, pending open-source artifacts and preferably a human/sandbox re-label check of the kind above. No stronger objection (e.g., circular evaluation or broken math) is required by the manuscript as written.","tokens_in":20608,"tokens_out":596,"duration_ms":6282,"concrete_test":"On a stratified sample of ~200 multi-skill (k≥2) test queries, have independent human experts (or a held-out agent sandbox) re-label joint executability of the GT set and of the top-m retrieved sets, blind to the original LLM SKIP labels. Report (i) agreement of human/sandbox C with LLM WRITE/SKIP and (ii) whether R3-Reranker’s Set-Compat lift over the no-SKIP reranker survives when Set-Compat is recomputed against the human/sandbox labels. If the lift collapses or LLM–human κ is low on forced_chaining / language_ecosystem_mismatch slices, the oracle premise fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim factors top-K skill correctness into relevance × C(q,S*_q) and treats DeepSeek-V4-Pro / Qwen3 WRITE–SKIP verdicts as the operational definition of C (§1.3, §2.1–2.4). Training SKIP labels are not exhaustively human-verified; only the test multi-skill GT intent is multi-expert reviewed, and Chinese skills remain sparse (246 raw). If those LLM judgments systematically diverge from real multi-skill executability (forced chaining, ecosystem conflicts, or sandbox install failures that an LLM never sees), then both the graded ListNet supervision and the Set-Compat gains measure recovery of the LLM’s own rejection surface rather than genuine compatibility. The paper already flags this in Limitations; it is the softest premise under the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that skill retrieval for LLM agents differs from document retrieval because top-K joint correctness depends on a query-conditioned compatibility factor C(q,S*_q) that independent per-skill relevance cannot recover. It constructs R3-Skill, a bilingual (Chinese–English) skill-routing benchmark with 10,246 skills, 41,592 WRITE queries, and 32,828 retained LLM SKIP annotations organized into an 8-class rejection taxonomy, with multi-expert verification of the test-set ground truth and a skill-pool-disjoint train/test split. On this data the authors train a two-stage system: R3-Embedding (multi-positive InfoNCE with a sibling-reward term on Qwen3-Embedding-0.6B) and R3-Reranker (graded ListNet with labels {3,1,0} for GT / SKIP partners / others on Qwen3-Reranker-0.6B). Appendix B derives bilateral balancing for bi-encoder SKIP injection (Theorem 1); ablations show SKIP hurts or fails to help the bi-encoder while helping the cross-encoder. The full pipeline reports Hit@1=0.7521, NDCG@10=0.8173, Set-Compat=0.3188 on R3-Skill and also leads on SkillRet transfer metrics.","tokens_in":20869,"tokens_out":1317,"duration_ms":10822,"significance":"If the results hold, the paper makes a concrete contribution to agent skill routing: it formalizes why skill retrieval is not document retrieval, releases a sizable bilingual resource that retains the usually discarded LLM rejection signal, and shows a practical two-stage training recipe with a clean dual-encoder gradient argument and stage-wise ablations. Strengths include the held-out skill-pool split, multi-expert test verification, shared-upstream reranker comparisons, SkillRet transfer, the Set-Compat metric, and the planned open release of data, weights, and scripts. These are useful for the growing ecosystem of agent skills (Anthropic Agent Skills, SkillRouter, SkillRet, etc.) where full-library prompting is infeasible and multi-skill compatibility matters.","major_comments":[{"comment":"§1.3 and §2.1–2.4 operationalize C(q,S*_q) via DeepSeek-V4-Pro / Qwen3 WRITE–SKIP joint-plausibility judgments; training SKIP labels are not exhaustively human-verified (only multi-skill test GT intent is multi-expert reviewed; Table 3). If those judgments systematically diverge from real multi-skill executability (sandbox install failures, ecosystem conflicts, or forced chaining an LLM never sees), then graded ListNet supervision and Set-Compat gains largely recover the LLM’s own rejection surface. The Limitations section already flags this; a load-bearing revision would add a human agreement study on a stratified SKIP sample (by direction and reject class) or a small end-to-end executability check, and report inter-annotator / LLM–human agreement so readers can bound oracle noise.","section":null},{"comment":"Table 9 and §4.4: zh2en (105) and zh2zh (92) are very small diagnostic slices, and Chinese skills are sparse (246 raw; §2.2, Limitations). Cross-lingual claims and the en2zh forced_chaining contrast are interesting, but the paper should either demote zh2* numbers to qualitative trends only (as partially done) or enlarge those slices / the Chinese skill pool before treating bilingual coverage as a primary contribution. Without that, the four-direction framing overstates the strength of the Chinese-side evidence relative to en2en/en2zh.","section":null},{"comment":"§4.1 and Appendix A define Set-Compat only on |GT|≥2 as 1 iff S*_q ⊆ top-m with m=|GT|. This is a useful set-level metric, but the manuscript should clarify how sensitive the reported 0.3188 is to the fixed reranker pool size (top-20) and to the graded labels {3,1,0}. A short sensitivity table (pool 10/20/50; label schemes binary vs graded) would show whether the Set-Compat gain is robust or an artifact of the engineering budget noted in Limitations (vi).","section":null}],"minor_comments":[{"comment":"Figure 1 caption reports p95=5,526 / p99=9,983 while the body text says p95=5,524 / p99=9,976; align the numbers.","section":null},{"comment":"§3: release checklist promises optimizer, LR, batch size, epochs, seed, hardware, and checkpoint rule for both models; ensure these appear in the camera-ready or appendix so the two-stage recipe is fully reproducible.","section":null},{"comment":"Table 5 note that R@K equals Comp@K on R3-Skill because |GT|≤3≤K is helpful; consider stating this once in Appendix A and dropping the redundant Comp columns on R3-Skill tables to reduce clutter.","section":null},{"comment":"Appendix D prompts are valuable; a short note on temperature / decoding settings used for DeepSeek-V4-Pro and Qwen3-235B would help others regenerate or audit the synthesis pipeline.","section":null},{"comment":"Typos / formatting: “Y outu Lab”, “difficult”, “official”, and occasional full-width punctuation in the PDF should be cleaned for the camera-ready.","section":null}],"recommendation":"major_revision","confidential_remarks":"The work is timely and the reject-as-resource idea is genuine. The main risk for a top IR venue is that the compatibility oracle is still largely LLM-defined; if the authors add even a modest human agreement / executability study on SKIP, I would lean accept after revision. Without it, major_revision is the right bar. Scope fit for cs.IR is good; novelty relative to concurrent SkillRouter / SkillRet is adequately disclosed in Table 4."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: skill top-K is not document top-K. Joint correctness needs a query-conditioned compatibility term that independent relevance cannot recover, and the usual LLM synthesis SKIP is a free signal for it if you stop throwing it away.\n\nWhat is actually new is the package. They keep 32k SKIP annotations, build an 8-class reject taxonomy, cover four Chinese–English directions with rewritten queries, hold out the skill pool, and multi-expert-check the test multi-skill GT. The formal split into relevance × C(q,S*) is clear, and Appendix B’s bilateral-balancing argument for why shared bi-encoder InfoNCE dilutes query-conditional SKIP is the cleanest piece of theory here. Ablations match it: SKIP hurts or fails the embedding stage and helps the graded ListNet reranker, especially Set-Compat. Transfer to SkillRet is real, not just home-field NDCG.\n\nSoft spots, in proportion. The load-bearing assumption is that DeepSeek/Qwen WRITE–SKIP is a faithful enough proxy for true multi-skill executability. Training labels are not exhaustively human-verified; Chinese skills are thin (246 raw); offline Hit/NDCG/Set-Compat are not tied to sandbox install success or task completion. The paper already says this in Limitations. That does not collapse the contribution—it means C is operationalized as the LLM’s rejection surface, and gains partly recover that surface. Still a better training signal than discarding SKIPs. Minor: zh2en/zh2zh are diagnostic slices, reranker pool fixed at top-20, free parameters (τ, λ, grades, KMeans) are standard IR knobs.\n\nMath and citation pattern look fine: InfoNCE + sibling term, ListNet grades, concurrent SkillRet/SkillRouter cited without pretending they do not exist. Data construction and the stage split are the real assets.\n\nThis is for people building agent skill routers and for IR folks who care about set-level, query-conditional negatives. I would bring it to reading group, cite the benchmark and the bi/cross split if I work on agent retrieval, and send it to peer review. Treat the oracle as provisional; open-source the artifacts.","headline":"Solid agent-IR systems paper: query-conditional skill compatibility is real, reject-as-resource is useful, and the bi-encoder vs cross-encoder split is cleanly argued—with the LLM oracle as the main provisional piece.","tokens_in":21507,"tokens_out":563,"would_cite":true,"duration_ms":6585,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Skill retrieval is not document retrieval: top-K success needs query-conditioned skill compatibility, recovered from LLM rejections usually thrown away.","keywords":["skill retrieval","LLM agents","query-conditioned compatibility","reject-as-resource","bi-encoder","cross-encoder reranker","bilingual benchmark","agent skill routing"],"falsifier":"A multi-expert or real-user study that systematically re-labels a large sample of multi-skill WRITE/SKIP decisions and shows that models trained on corrected labels reverse the reported Set-Compat gains, or that SKIP-as-graded-label no longer beats the no-SKIP reranker.","tokens_in":21461,"feed_emoji":"🧭","tokens_out":1009,"duration_ms":7585,"temperature":0.7,"pith_summary":"LLM agents compose reusable skills, so the front-end problem is retrieving the right set of skills for a user request. Document retrieval can treat relevant items as independent; skill retrieval cannot, because two skills that each look plausible may still conflict under a particular query—overlapping roles, mismatched ecosystems, or forced stacking. That query-conditioned compatibility factor is not recoverable from independent relevance scores alone. The paper shows that standard skill–query synthesis pipelines already produce a dense signal for it in the form of LLM WRITE/SKIP decisions, which are usually discarded. R3-Skill keeps those rejections as compatibility supervision, builds a bilingual Chinese–English benchmark with expert-checked test ground truth, and trains a two-stage retriever. Gradient analysis and ablations explain why the same signal is weak under a bi-encoder InfoNCE objective with bilateral balancing, yet useful as graded listwise labels in a cross-encoder reranker. The resulting pipeline improves joint set retrieval on the new benchmark.","feed_headline":"Skill retrieval needs compatibility, not just relevance","feed_subtitle":"LLM rejections usually discarded become the signal that keeps jointly usable skills together","key_machinery":"Reject-as-Resource (R3): retain LLM SKIP annotations as compatibility negatives, train R3-Embedding with multi-positive InfoNCE plus a sibling-reward term, then train R3-Reranker with graded ListNet labels (GT=3, SKIP partner=1, other=0) so the cross-encoder can enforce C-conditioned ranking.","core_discovery":"Top-K joint correctness in skill retrieval factors into per-skill relevance times a query-conditioned compatibility term C(q, S*_q) that independent relevance cannot recover. LLM WRITE/SKIP rejections supply usable binary supervision for C; injecting them as graded labels in a cross-encoder improves joint set metrics, whereas injecting them into the tested bi-encoder InfoNCE objective under bilateral balancing does not help.","pith_inferences":["Any retrieval setting where the returned set is jointly executed (tools, workflows, agents, code packs) may inherit the same relevance-times-compatibility decomposition, not only agent skills.","The 8-class rejection taxonomy suggests training separate heads or curricula for hard domain mismatch versus query-conditional forced chaining, rather than one global negative class.","Expanding the sparse Chinese skill pool and human-verifying training SKIPs would likely shift absolute numbers more than the bi-encoder vs cross-encoder split, which is structural.","End-to-end agent completion metrics may still diverge from Set-Compat if downstream planners re-order or drop retrieved skills; the paper’s offline gains need a handoff study."],"forward_implications":["Agent skill routers should treat joint set correctness (Set-Compat) as a first-class metric alongside Hit@K and NDCG, not as a side effect of independent relevance.","Synthesis pipelines that currently discard LLM rejections should keep them as compatibility supervision rather than pure low-quality waste.","Bi-encoders are a poor place to inject query-conditional compatibility under shared-vector bilateral balancing; cross-encoders can use the same signal as graded ranking labels.","Bilingual and cross-lingual skill routing (especially Chinese queries over English skill pools) needs explicit compatibility training, not only multilingual embedding alignment.","Sandbox and cold-start deployments that install only top-K skills per session become safer if the retriever respects query-conditioned non-co-occurrence."],"fun_headline_variants":["Skill retrieval needs query-conditioned compatibility not relevance alone","LLM rejections supply the missing skill compatibility signal","Independent relevance cannot recover joint skill-set correctness","Two-stage R3 retriever turns rejects into graded compatibility labels","Top-K skill routing factors relevance times query-conditioned C"],"cache_read_input_tokens":14592,"weakest_assumption_plain":"The load-bearing premise is that LLM WRITE/SKIP joint-plausibility judgments are a faithful enough stand-in for true skill compatibility under real user requests, even though training labels are not exhaustively human-verified.","fun_headline_variants_meta":{"raw":{"variants":["Skill retrieval needs query-conditioned compatibility not relevance alone","LLM rejections supply the missing skill compatibility signal","Independent relevance cannot recover joint skill-set correctness","Two-stage R3 retriever turns rejects into graded compatibility labels","Top-K skill routing factors relevance times query-conditioned C"]},"model":"grok-4.5","effort":"low","cost_usd":0.00666,"raw_usage":{"total_tokens":1788,"prompt_tokens":921,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":66600000,"prompt_tokens_details":{"text_tokens":921,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":805,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":921,"tokens_out":62,"duration_ms":7635,"temperature":1.0,"reasoning_tokens":805,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T15:14:27.914164+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A multi-expert or real-user study that systematically re-labels a large sample of multi-skill WRITE/SKIP decisions and shows that models trained on corrected labels reverse the reported Set-Compat gains, or that SKIP-as-graded-label no longer beats the no-SKIP reranker.","supporting_citations":[],"review_version":2}