{"id":"5ffe0f0c-c004-4d6b-91ee-48bab3b1854f","arxiv_id":"2605.24048","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Selecting LLM proposers by measured complementarity via greedy algorithms on small labeled data yields better performance-cost trade-offs than accuracy-only or diversity-only baselines in summarizer ensembles.","lead":"The paper reframes selecting which LLMs to include in an ensemble as a combinatorial complementarity problem and tests greedy selection algorithms on small labeled sets. A smart generalist might read it to understand practical ways to combine multiple AI models without running every possible combination.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Complementarity measured on small labeled set may not predict summarizer performance on unseen inputs","rationale":"The reader's weakest assumption directly identifies the same generalization risk that the argument depends on; the full-text placeholder does not supply additional safeguards (e.g., explicit cross-validation or distribution-shift experiments) that would remove the concern.","tokens_in":1643,"tokens_out":342,"duration_ms":22911,"concrete_test":"Reserve 30-50% of the labeled set as a held-out validation split never used for complementarity scoring; rerun the greedy selection on the remaining portion only, then measure end-to-end accuracy with the summarizer on both the original test set and the held-out validation split; if the selected subset's advantage shrinks by more than 2-3 points on the held-out split relative to the reported numbers, the generalization assumption does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on greedy selection algorithms that score proposer complementarity on a small labeled set and then deploy the chosen subset with a summarizer. For this to support the headline result, the complementarity metric (whatever its exact definition) must be stable under the distribution shift from the labeled set to the test inputs, and it must capture the relevant interactions with the particular summarizer model. The abstract and the referenced paragraph on greedy algorithms give no indication of held-out validation within the labeled set, cross-validation of the selection procedure, or explicit checks that the selected subset's advantage survives when the summarizer sees genuinely new data. If either condition fails, the reported performance-cost trade-offs are artifacts of the selection set rather than evidence for the guiding principle.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript reframes proposer selection for LLM ensembles (multiple proposers feeding a summarizer) as a combinatorial feature-selection problem that prioritizes complementarity among proposers and with the summarizer. It develops computationally tractable greedy-style selection algorithms that score complementarity on a small labeled set and reports experiments showing that this principle yields favorable performance-cost trade-offs compared with accuracy- or diversity-only baselines.","tokens_in":1799,"tokens_out":357,"duration_ms":10311,"significance":"If the complementarity metric is shown to be stable and predictive on held-out data, the approach could supply a practical, principled alternative to existing heuristics for constructing LLM ensembles, improving robustness without exhaustive search.","major_comments":[{"comment":"Abstract and paragraph on greedy-style selection algorithms: the central claim that complementarity measured on a small labeled set reliably predicts summarizer performance on unseen inputs is not supported by any description of held-out validation, cross-validation of the selection procedure, or explicit checks for distribution shift between the labeled set and test inputs. This is load-bearing for the headline result on performance-cost trade-offs.","section":"Abstract and paragraph on greedy-style selection algorithms"},{"comment":"Abstract: no information is given on how complementarity is quantified, what statistical tests or error bars accompany the reported trade-offs, or the data splits used, so the experimental validation of the guiding principle cannot be assessed from the provided text.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract could state the exact greedy algorithms considered and the size of the labeled set used for selection.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback emphasizing the need for clearer validation of the complementarity principle and additional experimental details. We address each major comment below.","responses":[{"response":"The current manuscript computes complementarity scores on a small labeled set to drive greedy selection and then evaluates the resulting ensembles on separate test inputs. However, it does not describe cross-validation of the selection procedure itself or explicit distribution-shift diagnostics. We agree these checks would strengthen the central claim and will add them (including k-fold validation of the selection step and shift analysis) in the revised manuscript.","revision_made":"yes","referee_comment":"[Abstract and paragraph on greedy-style selection algorithms] Abstract and paragraph on greedy-style selection algorithms: the central claim that complementarity measured on a small labeled set reliably predicts summarizer performance on unseen inputs is not supported by any description of held-out validation, cross-validation of the selection procedure, or explicit checks for distribution shift between the labeled set and test inputs. This is load-bearing for the headline result on performance-cost trade-offs."},{"response":"We will revise the abstract to state that complementarity is quantified via incremental performance gain on the labeled set, to note that reported trade-offs include error bars from repeated trials, and to specify the train/test splits employed.","revision_made":"yes","referee_comment":"[Abstract] Abstract: no information is given on how complementarity is quantified, what statistical tests or error bars accompany the reported trade-offs, or the data splits used, so the experimental validation of the guiding principle cannot be assessed from the provided text."}],"tokens_in":1266,"tokens_out":350,"duration_ms":20914,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point here is that the authors treat choosing which LLMs to feed into a summarizer as a combinatorial selection task and test several greedy algorithms that score complementarity on a small labeled set. This is not a new algorithm but a direct transfer of feature-selection ideas to the multi-LLM setting, and they report that some of the greedy variants give better performance-cost points than picking the strongest models or maximizing diversity alone.\n\nWhat the work does reasonably is lay out why pure accuracy or pairwise diversity can miss interactions with the summarizer, and it shows that a handful of computationally cheap selection rules can be run in practice. The experiments appear to compare multiple greedy heuristics and surface a few that look usable.\n\nThe soft spot is exactly the one flagged in the stress-test note. Complementarity is measured on the labeled set and then the selected subset is deployed; nothing in the abstract indicates held-out checks inside that set, cross-validation of the selection step, or explicit tests that the advantage survives when the summarizer sees new inputs. Without those, the reported trade-offs could be artifacts of the particular labeled examples rather than evidence that complementarity is a stable guiding principle. The abstract also gives no definition of the complementarity metric, no error bars, and no information on data splits or statistical tests, so the central empirical claim cannot be evaluated yet.\n\nThe paper is aimed at people who already run multi-model LLM pipelines and need a lightweight way to pick subsets. It is not advancing the theory of ensembles or feature selection. If the full manuscript contains careful validation that the selected subsets hold up on fresh data and includes the missing experimental details, it would be worth sending to referees; otherwise the contribution stays at the level of a practical heuristic note.","headline":"The paper applies standard greedy feature selection to LLM proposer choice and finds some practical trade-offs, but the generalization of the complementarity scores from the labeled set is not demonstrated.","tokens_in":2254,"tokens_out":426,"would_cite":false,"duration_ms":24721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Complementarity measured on small data guides better LLM proposer selection than accuracy or diversity alone.","keywords":["LLM ensemble","proposer selection","complementarity","multi-agent collaboration","greedy selection","summarizer model","performance-cost trade-off"],"falsifier":"Select a proposer subset using the complementarity method on one dataset, then measure whether that same subset still outperforms accuracy-based and diversity-based subsets when evaluated on a fresh, larger test distribution with the same summarizer.","tokens_in":2570,"feed_emoji":"🤖","tokens_out":659,"duration_ms":21728,"temperature":0.7,"pith_summary":"The paper treats the choice of which LLMs to include as proposers in an ensemble pipeline as a combinatorial feature-selection task, where an LLM's value depends on how its outputs complement the others and the downstream summarizer. Because full combinatorial search is too expensive, the authors test a range of greedy selection procedures that estimate complementarity from responses on a small labeled set. Experiments confirm that complementarity-driven selection yields stronger ensembles and better accuracy-cost trade-offs than baselines focused only on individual model strength or response variety. A reader would care because most current multi-LLM systems still pick proposers by crude heuristics; replacing those heuristics with complementarity checks offers a concrete, low-overhead improvement path.","feed_headline":"Complementarity beats accuracy or diversity for LLM ensemble selection","feed_subtitle":"Greedy algorithms on small labeled sets pick proposer combinations that improve accuracy while controlling total inference cost.","key_machinery":"Greedy-style selection algorithms that score complementarity from proposer responses on a small labeled set and feed the chosen subset into a fixed summarizer LLM.","core_discovery":"Proposer selection in LLM ensembles is best solved by measuring complementarity among proposers and with the summarizer on a modest labeled set and then applying computationally feasible greedy algorithms; this approach outperforms both strongest-model and diversity-only selection while controlling total inference cost.","pith_inferences":["The same complementarity principle could be tested in non-LLM multi-agent systems such as tool-use planners or debate frameworks.","If complementarity is stable across domains, one could maintain a reusable library of proposer subsets instead of re-running selection for every new task.","A natural next measurement is how quickly the complementarity ranking degrades as the labeled validation set shrinks below the sizes used in the paper."],"forward_implications":["Ensemble pipelines can achieve higher accuracy at lower total token cost by dropping redundant proposers.","The same selection logic applies to any fixed summarizer, allowing practitioners to swap in newer models without re-deriving the entire combination.","Performance gains are largest when the summarizer itself is weaker than the strongest proposers, because complementarity compensates for summarizer limitations.","The method remains practical even when the number of candidate LLMs grows, because only greedy passes over a small validation set are required."],"fun_headline_variants":["Complementarity outperforms accuracy and diversity for LLM ensembles","Greedy selection via complementarity refines LLM ensemble proposers","Small labeled complementarity guides optimal LLM ensemble selection","Combinatorial complementarity improves LLM ensembles at lower cost"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Complementarity scores computed on a small labeled set will continue to predict ensemble performance when the same proposers are used on new, unseen inputs.","fun_headline_variants_meta":{"raw":{"variants":["Complementarity outperforms accuracy and diversity for LLM ensembles","Greedy selection via complementarity refines LLM ensemble proposers","Small labeled complementarity guides optimal LLM ensemble selection","Combinatorial complementarity improves LLM ensembles at lower cost"]},"model":"grok-4.3","cost_usd":0.004562,"raw_usage":{"total_tokens":2230,"prompt_tokens":595,"num_sources_used":0,"completion_tokens":58,"cost_in_usd_ticks":45624500,"prompt_tokens_details":{"text_tokens":595,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1577,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":595,"tokens_out":58,"duration_ms":11410,"temperature":1.0,"reasoning_tokens":1577,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T16:45:35.838297+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Select a proposer subset using the complementarity method on one dataset, then measure whether that same subset still outperforms accuracy-based and diversity-based subsets when evaluated on a fresh, larger test distribution with the same summarizer.","supporting_citations":[],"review_version":1}