{"id":"8c39d20f-899d-4bf3-9f1d-8420dc34e170","arxiv_id":"2505.07251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Converting an m-class vision in-context learning task into m per-class yes/no questions improves accuracy when the retrieval database is missing some labels.","lead":"This paper studies what happens when the example database used for vision in-context learning is missing some categories, and proposes a simple fix: ask one yes/no question per category instead of one multi-class question. The fix, IJIP, improves accuracy on two benchmarks and also helps when the database is complete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"For missing labels, the binary sub-task has only negative demonstrations; the 'complete VICL' claim is therefore unsupported, and IJIP's gains may come from LVLM priors rather than from the proposed decomposition.","rationale":"The reader's weakest assumption identifies the same structural gap: the sufficiency of negative-only demonstrations for positive detection. My stress-test agrees and sharpens it into a concrete, testable claim. The concern does not by itself overturn the paper's empirical finding that IJIP scores well on two datasets and two LVLMs; those results are plausible even if the stated mechanism for missing labels is wrong, because the models may recognize CIFAR-10/Fashion-MNIST classes from pretraining. That is why I do not recommend changing the CONDITIONAL verdict: the paper should be asked to supply the zero-demonstration ablation for missing labels, to correct the overstatement in Section 3 and Section 4.1.1 ('always contains at least one instance of C_j'), and to report variance, since all numbers are three-run averages with no error bars. If the ablation shows no dependence on negative demonstrations, the central theoretical claim would need substantial revision, and a REJECT or stronger conditionality would be warranted. The empirical framing of the problem remains useful, and the proposed check is inexpensive and directly decisive.","tokens_in":16932,"tokens_out":3726,"duration_ms":39964,"concrete_test":"Ablate the retrieved demonstrations in the binary sub-prompts only for missing labels: for each j > w, query 'Is this C_j?' with zero in-context examples (or with random placeholder labels replacing the non-C_j labels), while keeping every other IJIP component identical, and measure accuracy on test images whose true label is among the missing labels. Repeat on CIFAR-10 and Fashion-MNIST with 10% missingness using InternVL 2.5-8B. If accuracy is statistically unchanged, the negative demonstrations are not performing the claimed work and the completeness argument is unsupported; if accuracy drops substantially, the mechanism is empirically validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conversion argument fails at exactly the point that motivates IICL. For a missing label C_j (j > w), the incomplete retrieval database contains no image labeled C_j, so the j-th binary sub-task can only supply C_j-bar (non-C_j) demonstrations. Yet Section 3 concludes that the sequence of binary tasks 'ensures a complete VICL process', and Section 4.1.1 states that 'the retrieval database always contains at least one instance of C_j'. Under the paper's own definitions in Eq. (1) and Eq. (4), both statements are false for j > w: the binary training set contains only the negative class. Standard in-context learning does not guarantee that seeing only 'Not C_j' examples enables a model to answer 'yes' when the input actually is C_j; at best the model can rely on its parametric knowledge of familiar classes such as CIFAR-10 and Fashion-MNIST. This is load-bearing because if negative-only demonstrations cannot drive positive identification, the binary stage contributes no information for exactly the classes that define the incomplete-database problem, and the claim that IICL is 'converted into a standard VICL scenario' collapses. The empirical results are consistent with this concern: zero-shot prompting alone achieves 86.7% on CIFAR-10, so strong LVLM priors could explain much of the missing-label performance without any genuine completion of the retrieval database.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the concept of Incomplete In-context Learning (IICL), where a vision-language model performs in-context learning with a retrieval database that lacks annotated examples for some classes. The authors propose IJIP, a two-stage framework: an Iterative Judgment Stage that reformulates the m-class problem into m binary sub-tasks (asking 'Is this C_j?' for each class), and an Integrated Prediction Stage that combines the binary predictions with a final multi-class in-context classification. The paper reports consistent accuracy gains over six VICL baselines across CIFAR-10 and Fashion-MNIST with InternVL 2.5-4B and 2.5-8B, at three label-missingness levels, and additionally applies IJIP to prompt learning and text classification. The central theoretical claim is that the binary decomposition converts IICL into a standard VICL scenario because each binary sub-task has demonstrations in at least one of its two classes.","tokens_in":17180,"tokens_out":5914,"duration_ms":56512,"significance":"The problem studied is well motivated and practically relevant: retrieval databases in dynamic settings are often incomplete, and existing VICL methods assume complete label coverage. IJIP is simple, training-free, and the empirical evaluation is broad for a short paper: two models, two visual datasets, three missingness levels, plus extensions to text and prompt learning. The main results are reproducible in the sense that accuracy numbers are reported for every condition and averaged over three runs, and the method contains no fitted parameters beyond the demonstration count k. The consistent gains over baselines, including the complete-database case, make the empirical contribution potentially useful. However, the paper's headline theoretical claim—that IICL is 'converted into a standard VICL scenario'—is not valid for the missing-label classes, because the binary sub-task for such a class contains only negative demonstrations. The contribution is therefore empirical rather than theoretical, and the theoretical framing must be revised to match the evidence.","major_comments":[{"comment":"The claim that the binary decomposition 'ensures a complete VICL process' is false for the classes that are missing from the retrieval database. Under the definition in Eq. (1), D_in contains only labels in {C_1,...,C_w}, so for any j > w the j-th binary demonstration set D^j_IJ in Eq. (4) contains only labels of the form non-C_j (i.e., C_j-bar). There is no positive example of C_j in the retrieved demonstrations. The statement in Section 3 that 'if j > w, at least C_j examples are available' is correct only if the overline is read as the negative class, and Section 4.1.1's assertion that 'the retrieval database always contains at least one instance of C_j' is contradicted by Eq. (1). A binary classification task with only negative demonstrations is not a standard complete VICL task; positive identification of C_j must come from the model's parametric prior, not from in-context demonstrations. This is load-bearing because the abstract and Section 3 use the 'complete VICL' argument to motivate IJIP. The authors should either weaken the claim to state that the binary sub-task always has demonstrations in at least one class, or add an experiment that directly tests whether negative-only demonstrations contribute to positive detection over a zero-shot control.","section":"Section 3 and Section 4.1.1"},{"comment":"The empirical results do not currently isolate the mechanism that the paper claims is responsible for the gains. For CIFAR-10 with InternVL 2.5-8B at 90% missing labels, IJIP achieves 89.2% while zero-shot prompting achieves 86.7%; the 2.5-point gap is small relative to the model's strong parametric prior on common classes like CIFAR-10. The paper does not report an ablation in which, for an available class, the positive demonstrations are withheld from the binary sub-task and the result is compared with the full-demonstration setting. Without such a control, the observed gains cannot be attributed to the proposed 'completion' of the retrieval database rather than to the model's prior knowledge or to the extra inference-time processing. This matters because the central claim of the paper is that IJIP mitigates the incomplete-database problem through the binary decomposition; a direct test of the negative-only demonstration regime is needed to support that claim.","section":"Section 5.2 and Table 2"}],"minor_comments":[{"comment":"Assumption 3.1 uses the norm \\|\\hat{y}_x - y_x\\|_p to measure the difference between predicted and true labels, but class labels are categorical and no embedding into a normed space is specified. Please replace this with a well-defined error measure, such as 0-1 loss, or define an explicit label encoding.","section":"Section 3, Eq. (6)"},{"comment":"The index range in the definition of D^j_IJ is written as '0 \\leq j \\leq m'; it should be '1 \\leq j \\leq m' because the sub-tasks are indexed by the class labels.","section":"Equation (4)"},{"comment":"The 'Reduction in Classification Complexity' paragraph asserts that binary tasks are 'inherently less complex' than an m-class task, but no complexity measure is defined and no evidence is provided. I recommend removing or substantiating this claim.","section":"Section 4.1.1"},{"comment":"The text refers to 'Table 6' in the first paragraph of Section 5.2, but the main results appear in Table 2. Please reconcile the table numbering throughout the manuscript.","section":"Section 5.2"},{"comment":"The overline notation for the negative class (non-C_j) is lost in the plain-text version, making the key argument in Section 3 difficult to follow. Please ensure the final typeset version renders \\bar{C}_j consistently and define it explicitly at first use.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The empirical study appears careful and the reported gains are consistent across settings, but the paper's central theoretical claim—that IICL is converted into a complete VICL problem—is not supported by the definitions or the experiments. I recommend requesting a revision that either reframes the contribution as empirical with an explicit negative-only-demonstration ablation, or substantially weakens the conversion claim. With such a revision, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2505.07251. The paper identifies a genuinely practical failure mode: in-context example databases are often incomplete, so retrieved demonstrations for missing classes are misleading. The fix, IJIP, is simple and easy to test: decompose m-class classification into m binary one-vs-rest sub-questions, then combine the answers. The decomposition itself is classic, but applying it to VICL under missing labels is new. The empirical results are consistent: IJIP beats six baselines across two LVLMs and two datasets at 10%, 40%, and 90% label missingness, and it even wins when the database is complete. The gain on Fashion-MNIST is large (zero-shot 37.2% to IJIP 78.9%), which shows the demonstrations are doing real work rather than the model's prior alone.\n\nThe soft spots are real but not disqualifying. The central conceptual claim — that binary decomposition 'converts IICL into a complete VICL scenario' — is overstated. For a missing class C_j, the j-th binary task has only 'not C_j' demonstrations, not positive examples. That is not a complete binary classification setting, and the model may be relying on its parametric knowledge of familiar classes. The 'less complex' argument is handwavy. The empirical appendix is thin: three runs averaged with no error bars, no released code, and only two small datasets. These are fixable in revision.\n\nThe stress-test worry goes further and says the method's advantage for missing labels might collapse because negative-only demonstrations cannot drive positive identification. The numbers do not support that: even at 90% missingness, IJIP beats the best baseline by a clear margin on Fashion-MNIST, and the zero-shot comparison shows a big improvement. So the empirical core holds up; the theoretical justification needs to be rewritten to say what negative-only demonstrations can and cannot do.\n\nWho should read this: anyone working on VICL, demonstration retrieval, or robust prompting. It is a useful problem definition plus a plug-and-play method, not a breakthrough. I would send it to a serious referee, with a request for code, variance estimates, a comparison with prior label-space decomposition work, and a corrected completeness claim.\n\nRecommendation: accept for peer review, expect revision.","headline":"Real problem, simple fix, consistent gains; the 'complete VICL' claim is overstated but the empirical core stands.","tokens_in":17706,"tokens_out":3270,"would_cite":true,"duration_ms":29954,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When a retrieval database lacks examples of some classes, decomposing the classification into per-class yes/no questions—so that every missing label still contributes \"not this class\" demonstrations—restores in-context learning accuracy…","keywords":["incomplete retrieval database","incomplete in-context learning","vision in-context learning","iterative judgments","binary classification decomposition","large vision-language models","prompt learning","missing labels"],"falsifier":"Run IJIP with a retrieval database that omits one class, then feed it only test images from the missing class and record the binary \"Is this C_j?\" answer for that class: if the positive rate is around chance, negative-only demonstrations are not doing the work and the gains must come from the second-stage fallbacks.","tokens_in":16725,"feed_emoji":"🧩","tokens_out":7552,"duration_ms":63556,"temperature":0.7,"pith_summary":"The paper introduces a previously unstudied failure mode of vision in-context learning: the retrieval database may contain labeled examples for only some of the classes in a classification task. It shows empirically that as more labels go missing, accuracy falls, though it stays above zero-shot prompting. The proposed remedy, IJIP, reformulates the m-class problem as m independent \"Is this class C_j?\" questions, relabeling every available image as either a positive or a negative for each sub-question. Because an image labeled with one class is implicitly a negative for all other classes, even a label with no examples still contributes \"not this class\" demonstrations, making the incomplete database behave like a complete one. The paper reports that IJIP beats all six baselines in nearly every setting, reaching 93.9% accuracy and working even when the database contains only one class.","feed_headline":"Ten yes/no questions beat one ten-way question when labels are missing","feed_subtitle":"Binary sub-questions turn an incomplete example database into a complete one, beating six baselines.","key_machinery":"The load-bearing mechanism is the binary reformulation plus the single combined prompt. Formally, a label $C_j$ is also an expression of $\\overline{C_i}$ for every $i\\neq j$; the method exploits this by turning one $m$-class question into $m$ sub-questions, each supplied with demonstrations relabeled from the original database into $\\{C_j, \\overline{C_j}\\}$. For $j\\le w$, both sides appear in the database; for $j>w$, the database contains only $\\overline{C_j}$ examples, yet the sub-question still has valid demonstrations. The paper claims this converts IICL into standard VICL. All $m$ sub-questions and their demonstrations are packed into one prompt, so the model is queried once in the first stage; the second stage uses the first stage's answer pattern to decide between direct assignment, full re-classification, or a restricted $u$-class re-classification.","core_discovery":"The central claim is that incomplete in-context learning can be repaired by changing the question format rather than by fixing the data. The paper's empirical study shows that IICL accuracy drops steadily as labels disappear from the retrieval database, and IJIP is designed to reverse that drop. In the Iterative Judgments Stage, one prompt asks the LVLM m binary questions about the same image, each with its own retrieved demonstrations whose labels have been converted to positive/negative for that class. For any class absent from the database, the available images from other classes still provide \"not this class\" demonstrations, so every binary task has both sides of the decision represented—at least negative evidence—and the problem becomes a complete VICL task. The Integrated Prediction Stage then decides the final label: if exactly one binary answer is positive, that label wins; if none are positive, a full m-class VICL inference is run; if several are positive, a smaller classification among only those candidates is run. The paper claims this yields the highest accuracy among six baselines under 10%, 40%, and 90% label missingness, that it holds even when all labels are present, and that it transfers to prompt learning and to text classification.","pith_inferences":["A testable generalization: IJIP suggests that any incomplete-label problem—class-incremental learning, extreme class imbalance, positive-unlabeled data—can be reduced to a set of binary tasks in which missing classes are represented only by negatives; the paper does not run these experiments.","The single-prompt design means the cost of asking m binary questions is close to the cost of one m-class question, so the approach could scale to very large label sets; the paper itself tests at most 24 classes (Wiki-Art Genre), leaving open whether long prompt contexts degrade the binary judgments.","If negative-only demonstrations are truly sufficient, then the model is doing something like implicit negative evidence: it learns \"not this\" from examples of other classes. A per-class binary accuracy breakdown would reveal whether missing classes are the weak point, which the paper does not report."],"forward_implications":["IJIP is plug-and-play: it can be layered on any VICL retrieval method, so existing systems can adopt the two-stage prompting without retraining or changing the database.","Because IJIP also beats all six baselines when no labels are missing, the decomposition itself—not just the missing-label fix—appears to help; this suggests the method is a general prompt-engineering improvement, not only a repair for incomplete data.","The extension to prompt learning (average +4.8%) and to text classification (SOTA on SST5 and Emotion) indicates the same binary-reformulation trick works when retrieval and prompting are done differently or in another modality.","The accuracy gain grows with LVLM size (up to 98.3% with the 26B model), so the benefit of the method scales with model capability rather than vanishing as the base model improves."],"supporting_citations":[{"why":"KATE provides the demonstration-retrieval approach that IJIP builds on and also serves as a baseline.","marker":"[22]"},{"why":"VICL-Rerank is a retrieval baseline that IJIP draws inspiration from and must outperform.","marker":"[48]"},{"why":"CLIP is the pre-trained encoder used to vectorize images for the similarity-based demonstration selection.","marker":"[35]"},{"why":"InternVL 2.5 is the vision-language model family on which the method is evaluated.","marker":"[7]"},{"why":"Clustering-retrieval is a baseline in the comparison set that IJIP claims to beat.","marker":"[21]"},{"why":"Cluster-Diversity is a baseline in the comparison set that IJIP claims to beat.","marker":"[31]"}],"fun_headline_variants":["Binary reformulation beats missing-label baselines in vision and text","Missing labels? Ask every class a yes/no with borrowed negatives","Splitting multi-label into binary fixes incomplete example sets","IJIP: Turn one m-way question into m yes/no to dodge missing labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method banks on a vision-language model being able to say \"yes, this is class C_j\" in a binary question even when every retrieved demonstration shows \"not C_j.\" If a model cannot transfer from purely negative demonstrations to a positive detection, the core advantage of IJIP for missing labels is lost.","fun_headline_variants_meta":{"raw":{"variants":["Binary reformulation beats missing-label baselines in vision and text","Missing labels? Ask every class a yes/no with borrowed negatives","Splitting multi-label into binary fixes incomplete example sets","IJIP: Turn one m-way question into m yes/no to dodge missing labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001379,"raw_usage":{"total_tokens":5650,"prompt_tokens":1077,"completion_tokens":4573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":4498}},"tokens_in":693,"tokens_out":4573,"duration_ms":27629,"temperature":1.0,"reasoning_tokens":4498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:20:57.776057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IJIP with a retrieval database that omits one class, then feed it only test images from the missing class and record the binary \"Is this C_j?\" answer for that class: if the positive rate is around chance, negative-only demonstrations are not doing the work and the gains must come from the second-stage fallbacks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KATE provides the demonstration-retrieval approach that IJIP builds on and also serves as a baseline."},{"cited_title":"Mot: Pre-thinking and recall- ing enable chatgpt to self-improve with memory-of-thoughts","cited_arxiv_id":null,"evidence_quote":"Clustering-retrieval is a baseline in the comparison set that IJIP claims to beat."},{"cited_title":"Diversity of thought improves reasoning abilities of large language models","cited_arxiv_id":null,"evidence_quote":"Cluster-Diversity is a baseline in the comparison set that IJIP claims to beat."}],"review_version":1}