{"id":"94a54d7a-da6f-4144-a473-68218a815f4b","arxiv_id":"2607.25880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A black-box fingerprinting method detects LLM provenance by measuring whether a suspect inherits the source's chosen answers on a curated set of multiple-choice questions.","lead":"This paper introduces Stemma, a black-box method that tests whether one AI model was derived from another by checking whether the two models give the same answers on specially selected multiple-choice questions. On 770 source-suspect pairs from 56 public checkpoints it reports 0.967 AUC, suggesting provenance can be detected from observable behavior alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark provenance labels are self-reported model-card claims; if any 'distillation'/'merge' pairs are not actually derived from the stated source, the headline AUC/TPR rest on corrupted ground truth.","rationale":"The reader's weakest-assumption analysis identified exactly the same load-bearing issue: provenance labels are derived from self-reported Hugging Face metadata and are not independently verified. I agree this is the most important threat to the central claim because it undermines the ground truth on which the entire evaluation rests. Other concerns — such as the reduced LLMPrint configuration or the absence of error bars — affect the strength of the comparison but not the validity of the provenance signal; the all-raw control experiment partly addresses baseline fairness, and stochastic variability is unlikely to flip a large AUC gap. By contrast, a few mislabeled 'distillation' checkpoints could directly contaminate the positive/negative split and inflate or deflate the reported AUC. The paper's limitations section acknowledges adaptive-attacker and interface-generalization concerns but does not acknowledge or test the provenance-label assumption, which is a notable gap. I therefore maintain the reader's CONDITIONAL verdict: the paper is well-executed and potentially important, but its headline claim should not be taken as established until the label audit described above is performed. Moving the verdict to ACCEPT or REJECT would be premature on the basis of this review.","tokens_in":24326,"tokens_out":7840,"duration_ms":77305,"concrete_test":"Manually audit all 42 derived checkpoints in Table 3: for each, inspect the Hugging Face model card, config fields (e.g., base_model, teacher_model, merge metadata, training scripts), and any released training documentation, and classify each checkpoint as 'confirmed derived from the stated source', 'confirmed not derived', or 'unknown'. Then recompute the main Table 1 metrics (AUC, pAUC, TPR@1%FPR) after moving confirmed-not-derived checkpoints to the negative pool and excluding unknown ones. If the headline numbers stay above roughly 0.9 AUC, the concern is empirically refuted; if they drop materially, the central claim must be scoped to only verified lineage labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that Stemma achieves 0.967 AUC and 87.8% TPR at 1% FPR — depends entirely on the correctness of the 770 pairwise labels, which are assigned from Hugging Face model-card descriptions, repository names, and tokenizer configurations (Appendix D, Table 3) without independent verification of training lineage. This is not a peripheral annotation issue: the fingerprint explicitly selects stable, non-gold decisions that a background model seldom reproduces, so whether a suspected checkpoint is truly derived from the source determines whether the measured agreement is evidence of provenance or merely shared behavior. Several listed 'distillation' entries are particularly risky: e.g., 'prithivMLmods/Regulus-Qwen3-R1-Llama-Distill-1.7B', 'eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b', and 'RedaAlami/Falcon3-7B-Instruct-Distill-DS-v1' have names implying a DeepSeek or Llama teacher, not the group's stated source; the paper does not demonstrate that these checkpoints are actually initialized from the claimed source. If even a few of the 98 positive pairs are not true derivatives — or if a cross-group pair is secretly related — the reported AUC, pAUC, and TPR are computed against mislabeled ground truth. The paper reports no error bars, no label audit, and no sensitivity analysis to label perturbations, so the headline numbers are only as trustworthy as the model-card claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces induced decision regions as a formalization for black-box LLM provenance testing. The key idea is to map open-ended model outputs into a finite decision space, instantiated here with multiple-choice questions, so that provenance is assessed by whether a suspect model preserves the source's decision-region assignments rather than by surface-form response similarity. Stemma operationalizes this by selecting probes on the source model that are stable under answer-option permutations, robust in log-probability margin, and specific against a background model, then scores a suspect by the alignment of source decisions across cyclic permutations. The authors report AUC 0.967, pAUC 0.944, and TPR 0.878 at 1% FPR across 770 source-suspect pairs from 56 public checkpoints, and 0.995 AUC on 1,260 deployment-variant pairs, outperforming four black-box baselines in the main setting. The paper includes extensive ablations on probe selection, pool size, probe budget, probe dataset, and background model, and releases an open-source implementation.","tokens_in":24745,"tokens_out":7536,"duration_ms":69026,"significance":"If the empirical claims hold, this is a useful contribution to black-box model provenance testing. The formalization is clean and it leads to a concrete, practical method with modest query cost (40 probes times 4 permutations) and strong separation in the evaluated benchmark. The paper is unusually thorough in its ablations, and the release of an open-source implementation is valuable for reproducibility. The main caveats are that the benchmark's ground-truth lineage labels are taken from model-card metadata without independent verification, and that the headline numbers are single-run point estimates. A control setting (all-raw prompts) also shows that one baseline (MPT) achieves higher overall AUC than Stemma when the prompt-format mismatch is removed, which tempers the claim of superiority over baselines. These issues are addressable with additional verification and reporting, but they affect the strength of the central empirical claim.","major_comments":[{"comment":"The provenance labels that define the 770 pairs are taken from Hugging Face model cards and repository names without independent verification. Several checkpoints labelled 'Distillation' appear to name a different teacher than the assigned group source (e.g., 'prithivMLmods/Regulus-Qwen3-R1-Llama-Distill-1.7B', 'eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b', 'RedaAlami/Falcon3-7B-Instruct-Distill-DS-v1'). If these positive pairs are not true derivatives, or if cross-group pairs are secretly related, the reported AUC/pAUC/TPR are computed against mislabelled ground truth. No label audit or sensitivity analysis to label perturbation is reported. Please verify lineage from README/training records or add a perturbation analysis and report metrics on a verified subset.","section":"§5.1 / Appendix D (Table 3)"},{"comment":"In the all-raw control setting, MPT achieves higher overall AUC (0.988 vs 0.965) and higher pretrained-source AUC (0.993 vs 0.968) than Stemma; Stemma has better pAUC (0.952 vs 0.938) and TPR at 1% FPR (0.898 vs 0.816). The manuscript describes Stemma as 'competitive' here, but this result directly qualifies the abstract's claim that Stemma 'substantially outperforms four representative baselines.' Because the all-raw setting removes the prompt-format mismatch that explains baseline failure in the main benchmark, this control should be reported prominently and the comparative claim should be rephrased or explained.","section":"§5.2 (Table 4)"},{"comment":"All headline results are single-run point estimates. Generation uses stochastic decoding (temperature 0.7, top-p 0.9), and Appendix I attributes small cross-run differences to stochastic decoding and hardware nondeterminism. No error bars, confidence intervals, or significance tests are reported for the 14-source comparisons, so it is unclear whether differences such as Stemma's 0.944 pAUC vs MPT's 0.613 (Table 1) are stable. Please report results over multiple seeds/fingerprint draws with intervals, or provide bootstrap CIs over source-suspect pairs.","section":"§5.2–§5.3 / Appendix I"}],"minor_comments":[{"comment":"Repository ID 'Qwen/Qwen3-1.7B' is listed with model type 'Instruct'; if this is intended to be the instruct checkpoint, the ID should be 'Qwen/Qwen3-1.7B-Instruct'. Please check that all repository IDs match the listed model types.","section":"Appendix D, Table 3"},{"comment":"The figure would benefit from explicit axis labels and a legend. The caption defines the green/orange coloring, but the rendered figure has no visible legend, making it hard to interpret without referring back to the caption.","section":"Figure 1"},{"comment":"The formatting of the MPT row is broken (e.g., 'MPT0.9930.946 0.878 3.258 0.991 0.972 0.939 3.333 0.9880.938 0.816 2.713'). Please fix the spacing and bold/underline markers.","section":"Table 4"},{"comment":"Treating invalid outcomes as non-aligned means that a suspect that systematically fails to produce parseable option labels will receive a lower score. This is a reasonable choice, but it should be stated as a limitation or at least discussed in the verification stage, since it interacts with prompt calibration quality.","section":"Equation (7)"},{"comment":"The exact filtering thresholds and ranking weights (0.5, 0, 0.5, 1.0, 0.5) are only given in prose. A small configuration table would improve reproducibility and make the method easier to reimplement.","section":"Appendix C"},{"comment":"The limitations discussion does not mention that the empirical evaluation is limited to models in the 1.7B–14B range. Given the paper's general title and conclusion, this scope restriction should be stated explicitly.","section":"§6 / Limitations"}],"recommendation":"major_revision","confidential_remarks":"The main risk to the paper's headline claims is the ground-truth lineage of the benchmark; the authors should be asked to provide independent verification or a label-perturbation analysis for the provenance groups. The all-raw control also needs to be framed honestly, because MPT's higher AUC there materially qualifies the 'substantially outperforms' claim. The formalization and method itself are sound enough to warrant a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read Stemma. The core idea is good: instead of comparing response surface, map outputs into a finite decision space (MCQ options) and treat provenance as inherited decision regions. That reframing is real, and probe selection via stability, robustness, specificity is a sensible, defensible package. The experiments are extensive—7 provenance groups, 56 checkpoints, deployment variants, ablations, and they open-source the code. The deployment-robustness results are a nice addition. I believe the central claim that this signal is stronger than response-level similarity.\n\nThe soft spots. The biggest is the provenance labels. Positive pairs are assigned from Hugging Face model-card claims. No independent verification. The stress-test note names specific distillation repos where the model name suggests a different teacher than the group's source. I can't vouch for those specific repos, but the point stands: if a few positive pairs are not actually derived from their claimed source, your AUC/TPR are computed against corrupt ground truth. The paper gives no sensitivity analysis to label errors. This is not fatal—the effect on 770 pairs might be small—but it's a real gap.\n\nSecond, no error bars or repeated seeds on headline numbers. The ablation sweep shows moderate variance across pool sizes, so a single number without confidence is less reassuring than it could be.\n\nThird, the baseline comparison. In the all-raw control, MPT hits 0.988 AUC vs Stemma 0.965, though Stemma wins at low-FPR. The paper says this openly. LLMPrint is reduced to 200 GCG steps vs the original, which may handicap it. So \"substantially outperforming\" applies to the main setting, not uniformly.\n\nNone of these are killers. The method is clean, the writing is honest, and the authors acknowledge the adaptive-attacker limitation and the MCQ-only interface. This is a serious contribution to the fingerprinting literature.\n\nMy recommendation: send it to peer review. It's the kind of paper referees can push on. Ask for label validation or at least a perturbation analysis of the benchmark labels, and for error bars on the headline. But the core idea and empirical package warrant referee time.","headline":"Stemma is a genuinely useful provenance-fingerprinting paper whose headline results are strong but rest on unverified model-card lineage labels; worth serious review with a demand for label validation.","tokens_in":25164,"tokens_out":2194,"would_cite":true,"duration_ms":22081,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model's private decision regions — the choices it makes when responses are mapped to a finite answer set — are inherited by its fine-tunes, adapters, merges, quantisations, and distillations, and this inheritance lets a black-box","keywords":["LLM provenance","model fingerprinting","induced decision regions","decision region inheritance","black-box verification","model lineage","multiple-choice probing","provenance testing"],"falsifier":"Take model pairs whose derivation is certified by independent training logs or weight-difference analysis rather than model cards, hold out one model family, and measure Stemma's alignment scores on those certified pairs: if AUC drops well below 0.967, the headline number rests on unverified labels. Alternatively, fine-tune a suspect to minimise agreement on the source's recorded non-gold probes and check whether the alignment score falls; if it stays high, the non-adaptive assumption is doing the work.","tokens_in":24252,"feed_emoji":"🧬","tokens_out":6866,"duration_ms":56847,"temperature":0.7,"pith_summary":"The paper argues that LLM provenance can be tested by comparing 'induced decision regions' rather than surface responses: map each open-ended output into a finite decision space (here, multiple-choice answer options) and record which region each query falls into. Related models inherit the source's region assignments more strongly than unrelated models, so the degree of inheritance becomes a black-box signal of shared lineage. Stemma operationalises that signal by selecting probes whose source decisions are stable across option permutations, robust against boundary shifts, and specific against unrelated background models, then scoring a suspect by the fraction of probe–permutation observations that preserve the source's recorded decisions. Across 770 source–suspect pairs from 56 public checkpoints, the method reports 0.967 AUC and 87.8% true-positive rate at 1% false-positive rate, and 0.995 AUC on 1,260 pairs spanning 91 deployment instances. If correct, this gives practitioners a practical, query-only way to link a suspect API or checkpoint to a source model.","feed_headline":"Decision-region fingerprints trace LLM lineage at 0.967 AUC","feed_subtitle":"Even after fine-tuning, merging, or quantization, a suspect model's answer choices give away its source model.","key_machinery":"Induced decision regions: for a finite decision space D (here, the answer options of multiple-choice questions), a model's discrete decision map h_M(q) assigns each query to a region R^M_d, partitioning the query space into a decision geometry. The carrier of the argument is the empirical claim that this geometry is inherited — descendants of a source preserve region assignments more often than unrelated models, even when response surface forms shift. Three probe-selection principles make the inheritance observable: stability (the source decision persists across cyclic option permutations), robustness (the decision sits well inside its region, with a large log-probability margin over the run","core_discovery":"The central claim is that induced decision region inheritance is a reliable provenance signal. For a model M and finite decision space D, the discrete decision map h_M(q) partitions the query space into induced decision regions R^M_d = {q : h_M(q)=d}; provenance is measured by how much of S's region geometry survives in T. Stemma makes this observable with a finite probe set: it calibrates a prompt template, selects questions whose source decisions are stable under cyclic option permutations, lie well inside their region, and disagree with both the gold answer and a background model, then computes fingerprint alignment A(S,T), the fraction of probe–permutation observations that preserve the","pith_inferences":["The paper instantiates induced decision regions with multiple-choice questions, but the formalism only requires a mapping from open-ended outputs to a finite decision space; refusal/escape categories, classification-style outputs, or verifier-defined behavioural tests are natural extensions that should carry the same inheritance signal.","The deliberate focus on stable, non-gold, background-specific decisions suggests that a model's idiosyncratic confident errors are its most diagnostic fingerprints — a property worth testing on privately constructed or adversarially assembled question pools, where an adaptive suspect would not know the probes.","If the reported persistence holds across families, provenance scores could be paired with licensing audits: a small set of source-elicited probes queried through a suspect API would give statistical lineage evidence even when the suspect is served behind a deployment wrapper or retrieval pipeline.","The stated non-adaptive threat model leaves a direct test standing: fine-tune a suspect to avoid agreement on the public probe distribution and measure how much the alignment score drops; the authors note this risk, and the outcome would bound the method's real-world evasion resistance."],"forward_implications":["Provenance testing can be conducted with black-box query access only — no weights, activations, logits, training data, or development history from the suspect.","The signal survives interface mismatch: source fingerprints created under raw completion prompts still separate suspects queried through chat templates, where response-based baselines degrade toward random.","The method stays near-perfect across deployment variations — system prompts, role-play, chain-of-thought, retrieval augmentation, and decoding changes — because decisions abstract away surface form.","Ablations show the selection principles matter: random probes give 0.751 AUC, the full stability–robustness–specificity ranking reaches 0.967 AUC, with pAUC jumping from 0.614 to 0.944.","One-time fingerprint construction takes roughly 4.5 minutes and verification 38 seconds, making the approach practical for audits of many suspects."],"fun_headline_variants":["Stemma: decision-region fingerprints identify LLM lineage","Decision-region inheritance reveals an LLM's source","Stemma traces LLM lineage by decision-region inheritance","Induced decision regions fingerprint LLM provenance","0.967 AUC: decision regions reveal LLM ancestry"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's provenance labels are taken at face value from public model-card documentation — every checkpoint labelled a fine-tune, adapter, merge, quantisation, or distillation of a source is genuinely derived from that source, and cross-group checkpoints are genuinely unrelated; if any label is wrong, the reported AUC values partly measure mislabelled pairs.","fun_headline_variants_meta":{"raw":{"variants":["Stemma: decision-region fingerprints identify LLM lineage","Decision-region inheritance reveals an LLM's source","Stemma traces LLM lineage by decision-region inheritance","Induced decision regions fingerprint LLM provenance","0.967 AUC: decision regions reveal LLM ancestry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00089,"raw_usage":{"total_tokens":3680,"prompt_tokens":755,"completion_tokens":2925,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2856}},"tokens_in":499,"tokens_out":2925,"duration_ms":17964,"temperature":1.0,"reasoning_tokens":2856,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:12:19.851106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take model pairs whose derivation is certified by independent training logs or weight-difference analysis rather than model cards, hold out one model family, and measure Stemma's alignment scores on those certified pairs: if AUC drops well below 0.967, the headline number rests on unverified labels. Alternatively, fine-tune a suspect to minimise agreement on the source's recorded non-gold probes and check whether the alignment score falls; if it stays high, the non-adaptive assumption is doing the work.","supporting_citations":[],"review_version":1}