{"id":"e79f5ef1-a941-47bb-9759-00f1dcf00246","arxiv_id":"2412.01443","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A synthetic-data augmentation method that decomposes documents into facets and recombines LLM-written similar and dissimilar fragments improves faceted query-by-example retrieval without citation labels.","lead":"Researchers built a method that breaks documents into facets, asks a language model to generate similar and different versions of each facet, and recombines them to create training data for facet-aware retrieval. The method improves retrieval on scientific abstracts and introduces a benchmark for educational exam questions, targeting domains where citation labels are unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sign of the LLM-generated positive/negative facets is never validated; without a label-swap control, the reported gains may reflect generic in-domain triplet fine-tuning rather than facet-conditional semantics.","rationale":"The reader identifies LLM pseudo-label validity as the weakest assumption; I agree and sharpen it. The paper's own evidence for the causal role of facet semantics is only qualitative (Fig. 3) and the MiniLM step governs only negative hardness, not the correctness of the positive/negative sign. Since the training documents come from the same broad domain as the CSFCube test set, additional contrastive fine-tuning on synthetic paper-like text could by itself yield gains. A label-swap experiment is the minimal control that separates 'the LLM's similarity judgment is informative' from 'any coherent generated text plus triplet loss helps.' The reader's CONDITIONAL verdict already expresses this kind of concern, so my read does not change the verdict; the requested control would determine whether the condition is actually met.","tokens_in":16079,"tokens_out":6295,"duration_ms":58734,"concrete_test":"Run the label-swap control on CSFCube: fine-tune SPECTER on the exact FaBle-generated pairs used for Table 2, but define the positive as [C^f_dis; ...] and the negative as [C^f_sim; ...] for every triplet. If the swapped model reproduces the reported NDCG%20/MAP gains, the augmentation's benefit is independent of the LLM's similar/dissimilar judgments and the facet-conditional interpretation of the central claim fails. If swaps degrade performance, the sign of the pseudo-labels is informative. Report mean and std over the same three seeds; also report the same per-query improvement proportions as in the Limitations section for a direct comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FaBle's central claim is that 40.68K synthetic triplets built from 1K abstracts teach SPECTER facet-conditional embeddings (abstract; §5). The learning signal in Eqs. (3)–(4) reduces to the sign of the generated target facet: D_f+ contains C^f_sim, D_f- contains C^f_dis. Both fragments are produced by LLaMA2-13B from the same summary S_f using only the prompt variants 'write a similar {facet}' vs 'write a dissimilar {facet}' (§3.2, Fig. 2). There is no human or independent check that C^f_sim is actually more relevant to the query under facet f than C^f_dis; Fig. 3 shows two examples, and the MiniLM rescoring in §3.5 filters only negatives. If the sign is noisy or arbitrary, the contrastive loss is still well-defined and can improve in-domain retrieval by providing 40K paper-text pairs for fine-tuning, especially because the 1K S2ORC abstracts are drawn from the same CS domain as CSFCube. The ablation in Table 4 replaces negative fragments with random original facets, but it never swaps positive and negative labels or replaces C^f_sim with a random facet, so it does not isolate whether LLM similarity judgments—rather than any syntactically coherent generated text—drive the gains. This is the load-bearing unvalidated assumption in the causal reading of the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FaBle, a fully synthetic augmentation method for faceted query-by-example retrieval. The method decomposes each document into facet summaries using zero-shot prompting of LLaMA2-13B, generates facet-specific 'similar' and 'dissimilar' fragments by recursive prompting, and recomposes these fragments into positive and negative pseudo-documents. The authors fine-tune SPECTER on the resulting triplet pairs and evaluate on CSFCube, where they report aggregate NDCG%20 and MAP gains over the base SPECTER, and on a newly released FEIR benchmark for TOEFL-style exam items, where gains are reported for all facets. The paper also includes an ablation replacing generated negatives with random facets and a hard-negative regeneration step based on MiniLM scores.","tokens_in":16426,"tokens_out":8833,"duration_ms":74521,"significance":"If the reported gains hold up, the paper's contribution is a low-resource route to facet-conditional embeddings: it uses only 1K documents and no citation labels or manual facet annotations, and it provides the first faceted QBE test set in the education domain. The release of code, datasets, and the FEIR benchmark, together with the query-level win-rate analysis and random-negative ablation, are concrete strengths. The main caveat is that the central causal claim—that LLM similarity judgments teach true facet semantics—rests on an assumption that is asserted but not validated; the label-sign is never tested against an independent signal. The absence of deduplication between the S2ORC training sample and the CSFCube test set and the score-variance-based selection of FEIR queries add further uncertainty.","major_comments":[{"comment":"The label-sign of the LLM-generated positives and negatives is never independently validated. The only distinction between D_f+ and D_f- is the prompt phrasing 'write a similar {facet}' versus 'write a dissimilar {facet}', both conditioned on the same LLaMA2-13B summary S_f. The FaBle-RN ablation in Table 4 replaces only the negative fragments with random original facets; it does not swap the labels or replace C_f^sim with a random facet. Consequently, the observed gains are also consistent with generic in-domain triplet fine-tuning on 40K syntactically coherent paper-text pairs, rather than with the facet-conditional semantics claimed in the abstract. I ask for two control conditions: (i) train with the labels flipped (C_f^dis as positives and C_f^sim as negatives), and (ii) train with random original facet summaries as positives while keeping the generated negatives; if either control also improves over the SPECTER baseline, the causal interpretation in the abstract is not supported. In addition, the assertion in §3.1 that S_f 'highly represents the facet' is supported only by qualitative examples in Figure 3 and by full-document similarity in Table 3, neither of which measures facet fidelity against a reference.","section":"§3.2, Eqs. (3)–(4); §7, Table 4"},{"comment":"The 1,017 S2ORC abstracts are selected from the computer science domain, the same domain as the CSFCube test collection, but the paper does not report whether these abstracts are deduplicated against the CSFCube query or candidate papers. Because FaBle constructs pseudo-documents from the full text of these abstracts, any overlap between the augmentation source and the test collection could inflate the reported gains. Please report the exact number of overlapping paper identifiers between the S2ORC sample and the CSFCube papers, and if the count is nonzero, re-run the experiments after removing the overlapping abstracts.","section":"§5, Data and Settings"},{"comment":"The FEIR test queries are selected by taking the eight items with the largest standard deviations of MiniLM similarity scores, and the same MiniLM cross-encoder is used in the hard-negative generation pipeline of §3.5. This means the new benchmark is intentionally biased toward items that MiniLM-style representations find heterogeneous, which may make the test set unrepresentative of normal exam-item retrieval and may align with the inductive biases of the augmentation procedure. Please justify this selection criterion with respect to the intended use of FEIR, and report results under a random selection of queries or across multiple selection seeds to show the benchmark is not tuned to the method.","section":"§4, Dataset Construction"},{"comment":"The paper does not provide significance tests, and the Limitations section states that no statistical tests are provided due to the small test-set size. Several facet-level comparisons move in the opposite direction (e.g., Background MAP decreases from 43.95 to 42.66 in Table 2), and the standard deviations overlap for some gains. Since the abstract uses 'remarkably' and the text uses 'significantly' for the method facet, the central claim should be supported by bootstrap confidence intervals or paired significance tests across the 50 CSFCube and 24 FEIR queries, or by clearly presenting the query-level win rates already reported in Limitations as the main robustness evidence.","section":"Limitations; Tables 2 and 5"}],"minor_comments":[{"comment":"'Fable' is a typo for 'FaBle', and the sentence 'Fable explicitly create' has a subject-verb agreement error.","section":"§1, p. 2"},{"comment":"The header 'CFSCUBE' should be 'CSFCube'.","section":"Table 2"},{"comment":"The pair-count derivation is unclear: after stating that four D_f+ and four D_f- are generated per document, the text says 'five documents, including the original one, lead to ten (D_f;Q, D_f+) pairs (i.e., five choose two)', which does not specify which documents are anchors and which are positives. Please rewrite the counting argument with explicit per-facet totals.","section":"§3.3"},{"comment":"The text says NDCG%20 is computed at 20% of the query pool size, but the formula K = p*|C| uses the candidate pool size; please correct the wording.","section":"§5, Evaluation Metrics; Appendix D"},{"comment":"The Ethical Statement says 'two human experts', while §4 says three experts were hired (a professor and two English specialists); please reconcile the numbers.","section":"Ethical Statement"},{"comment":"The prompt text in Figure 2 is too small to read in the PDF; please provide the full prompts in a table or appendix.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped empirical paper with a useful new benchmark, but the missing label-swap control and the lack of deduplication are the two points I would require before accepting. The FEIR construction procedure also deserves scrutiny; if the benchmark is intended as a shared resource, the variance-based query selection should be documented as a deliberate design choice rather than an incidental artifact. I would be willing to accept after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the decompose-recompose idea is genuine, and the FEIR benchmark is a useful addition, but the abstract's causal claim — that these are facet-conditional embeddings — is not isolated by the experiments. A label-swap or positive-replacement control is missing.\n\nWhat's new: instead of relying on citations or expensive ChatGPT annotation, they use LLaMA2-13B to summarize each facet, then generate 'similar' and 'dissimilar' facet fragments, then concatenate into pseudo-documents for triplet training. That's a clean, cheap recipe. The FEIR test set, built from TOEFL-QA with expert annotation and reported inter-annotator agreement, fills a real gap in educational item retrieval. The paper is clearly written, the prompting details are reproducible, and the limitations section is honest (no statistical tests, per-query improvement proportions instead).\n\nThe results are consistent: SPECTER+FaBle beats SPECTER on CSFCube overall, and the largest gain is on the method facet, which is plausibly the most facet-dependent. On FEIR, gains appear on all facets. Those within-model comparisons support the practical value of the augmentation.\n\nThe soft spots, in order of size. First, the load-bearing assumption: the entire training signal is the LLM's own distinction between 'similar' and 'dissimilar' facet fragments. There is no human or independent check that the positives are actually more facet-relevant than the negatives. The MiniLM rescoring only filters easy negatives; it does not validate the sign. The ablation swaps negatives for random facets but never swaps the positive, so it doesn't rule out that any fluent in-domain triplet text — not facet-conditionality — drives the gains. A simple label-swap control would settle this. Second, the 1,017 S2ORC abstracts are from the same domain as CSFCube and no dedup is reported; some leakage could contribute. Third, the FEIR queries were selected by maximum score variance, which is defensible but not a random sample. Fourth, 'no pre-defined facet knowledge' is overstated: the facet names (background/method/result) are pre-specified.\n\nNone of this kills the paper. The method is plausible, the benchmark is reusable, and the per-facet breakdown on CSFCube method is suggestive. But the central causal reading should be fixed before publication.\n\nWho's it for: IR people working on low-resource retrieval or LLM-based augmentation, and anyone who wants an education-domain faceted QBE test set. I'd send it to review — the concerns are addressable with one or two additional experiments, not a rework.","headline":"Solid augmentation recipe and a useful new benchmark, but the facet-conditional claim rests on an untested sign of LLM-generated triplets.","tokens_in":16887,"tokens_out":3852,"would_cite":false,"duration_ms":31559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Faceted query-by-example retrieval can be trained without facet-level labels by having a language model decompose documents, generate facet-specific similar and dissimilar fragments, and recompose them into synthetic training triples.","keywords":["faceted query-by-example","data augmentation","LLM-generated training data","contrastive learning","scientific paper retrieval","educational item retrieval","facet decomposition","retrieval benchmark"],"falsifier":"Take the 1K abstracts, have human annotators mark which sentences realize each facet, and measure overlap with LLaMA2's generated summaries; low agreement would mean the synthetic pairs encode the model's bias rather than facet-level relevance, so the reported gains would not transfer to new domains.","tokens_in":15904,"feed_emoji":"🔍","tokens_out":7384,"duration_ms":57265,"temperature":0.7,"pith_summary":"The paper claims that fine-grained 'query by example' retrieval—finding documents similar to a query document under a specified facet such as background, method, or result—can be trained without any facet-level relevance labels. The proposed FaBle method lets an open language model decompose each document into facet summaries, generate facet-matched and facet-mismatched fragments, and recompose those fragments into synthetic positive and negative training documents. Fine-tuning a document encoder on these synthetic pairs improves retrieval on the CSFCube scientific-paper benchmark, with the largest gains on the method facet, and also improves retrieval on a new educational exam-item benchmark, FEIR, introduced by the paper. The paper's bottom line is that modular synthesis, rather than citation labels or human annotations, can supply the facet signal that current methods lack.","feed_headline":"Synthetic facet pairs lift faceted retrieval from just 1K documents","feed_subtitle":"Fine-tuning SPECTER on LLM-built facet positives and negatives beats citation-trained models on the method facet.","key_machinery":"The load-bearing mechanism is multi-facet blending: a three-stage pipeline that decomposes (prompts LLaMA2-13B to summarize each facet), generates (self-feeds the facet summary into prompts asking for a similar and a dissimilar facet fragment), and recomposes (concatenates the generated fragments to form facet-conditioned positives and negatives, and optionally rescores negatives with a MiniLM cross-encoder to create hard negatives). This machinery turns one unlabeled document into forty triplet pairs per facet, so roughly 1K abstracts yield 40.68K training triples.","core_discovery":"The central discovery is that explicit facet-conditioned training pairs can be synthesized without labels: a small open LLM (LLaMA2-13B) summarizes each facet of an abstract, generates a similar version of that facet and a dissimilar version on a different topic, and the recomposed combinations serve as positive and negative documents for contrastive training. On CSFCube, applying this augmentation to SPECTER yields average gains of 3.4 NDCG@20 and 1.4 MAP, with the method facet gaining 7.6 NDCG@20 and 3.5 MAP; with the co-citation-trained SPECTER-COCITE base, FaBle outperforms ASPIRE models trained on roughly 32 times more data. The paper also claims domain transfer: on FEIR, built from TOEFL listening items, every facet improves after FaBle augmentation.","pith_inferences":["If the facet summaries are faithful, the same decomposition could serve as a reusable pre-training signal: any corpus with a named facet taxonomy could be augmented without human labels, not only abstracts and exam items.","The hard-negative rescoring step suggests a testable extension: replacing the MiniLM scorer with a prompt asking the LLM to self-score generated fragments might make the pipeline fully self-contained.","A direct comparison against training on human-annotated facet pairs of the same size would isolate how much of the gain comes from the synthesis itself rather than from the contrastive objective.","The FEIR benchmark, with eight queries per facet, is small, yet it gives the education domain a first public testbed for facet-conditioned retrieval."],"forward_implications":["Fine-tuning on FaBle pairs improves every facet of CSFCube over SPECTER, with the largest effect on the method facet, which prior models handle worst.","The same augmentation transfers to a new domain: on the released FEIR benchmark built from TOEFL items, all facets improve, with the largest gains on the question facet.","Because the pipeline needs no citations or facet labels, it opens faceted QBE to domains like education and law where such labels do not exist.","Increasing the FaBle-augmented corpus from 0.5K to 1K documents consistently raises NDCG@20 and MAP, especially for the method facet.","Hard negatives generated by score-controlled regeneration help the method facet but not background or result, suggesting the augmentation can be tailored per facet."],"supporting_citations":[{"why":"Introduces SPECTER, the document embedding model that FaBle fine-tunes and the main baseline for faceted QBE.","marker":"Cohan et al., 2020"},{"why":"Provides the CSFCube test collection and the 0-3 relevance annotation schema used to evaluate faceted retrieval.","marker":"Mysore et al., 2021"},{"why":"Provides the ASPIRE models and co-citation training data that serve as strong comparison baselines.","marker":"Mysore et al., 2022"},{"why":"Supplies LLaMA2-13B, the open language model used for facet decomposition and fragment generation.","marker":"Touvron et al., 2023"},{"why":"Supplies MiniLM, the cross-encoder that scores generated negatives for hard-negative regeneration.","marker":"Wang et al., 2020"},{"why":"Supplies the S2ORC corpus from which the 1K unlabeled paper abstracts are drawn.","marker":"Lo et al., 2020"},{"why":"Provides the TOEFL-QA dataset from which the new FEIR benchmark is constructed.","marker":"Chung et al., 2018"},{"why":"Introduces the TOEFL listening comprehension items that FEIR adapts into exam-item queries and candidates.","marker":"Tseng et al., 2016"},{"why":"Provides MS MARCO, the training data for the MiniLM cross-encoder used in hard negative scoring.","marker":"Bajaj et al., 2016"}],"fun_headline_variants":["LLM-built facet pairs lift retrieval without labels","Synthetic facet training beats citation-based models","FaBle augments 1K docs for faceted QBE gains","Decompose and recompose: LLM makes facet pairs","Small open LLM synthesizes training data for facets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that the language model's automatic summaries and generated snippets isolate the requested facet the way a human would, because those snippets are the only training signal.","fun_headline_variants_meta":{"raw":{"variants":["LLM-built facet pairs lift retrieval without labels","Synthetic facet training beats citation-based models","FaBle augments 1K docs for faceted QBE gains","Decompose and recompose: LLM makes facet pairs","Small open LLM synthesizes training data for facets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1508,"prompt_tokens":918,"completion_tokens":590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":534,"tokens_out":590,"duration_ms":5614,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:22:54.903404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 1K abstracts, have human annotators mark which sentences realize each facet, and measure overlap with LLaMA2's generated summaries; low agreement would mean the synthetic pairs encode the model's bias rather than facet-level relevance, so the reported gains would not transfer to new domains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CSFCube test collection and the 0-3 relevance annotation schema used to evaluate faceted retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TOEFL-QA dataset from which the new FEIR benchmark is constructed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the TOEFL listening comprehension items that FEIR adapts into exam-item queries and candidates."}],"review_version":1}