{"id":"24d5097e-0d14-4a85-8ece-a7c27b865d87","arxiv_id":"2606.23403","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Litmus creates zero-label metrics from code analysis for AI evaluation, achieving broader coverage, lower redundancy, and higher validity than baselines on three pipelines including scientific QA with Spearman ρ=0.72.","lead":"Litmus is a system that automatically designs evaluation metrics for AI pipelines by analyzing their source code and targeted questions without using any labeled data. A smart generalist might read it to see a potential shift toward making AI evaluations more explicit and tied to actual system intent rather than ad-hoc choices.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Targeted interrogation likely requires external knowledge of intended goals, undermining the zero-label claim","rationale":"The reader's weakest_assumption is precisely the load-bearing point; the abstract provides no counter-evidence that interrogation avoids goal knowledge. Because the full text was unavailable to the reader, the concern cannot be resolved from the given material, so the UNVERDICTED / LOW verdict stands. No other internal inconsistency (e.g., in the reported Spearman numbers or portfolio properties) rises to the same level of centrality.","tokens_in":1840,"tokens_out":375,"duration_ms":17586,"concrete_test":"Extract the exact interrogation protocol, prompt templates, and any example dialogues from the methods section; classify each question as (a) purely syntactic/code-derived or (b) presupposing semantic knowledge of pipeline goals or failure semantics. If more than 30% fall into (b), re-run the three-pipeline comparison after removing those questions and check whether the validity and coverage advantages persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Litmus identifying 'what must be measured and why' from source code plus targeted interrogation in a strictly zero-label regime with no external knowledge of goals. The load-bearing assumption is that this elicitation process can succeed without labels or goal knowledge. Targeted interrogation, however, is the mechanism for surfacing intent; any effective interrogation protocol must pose questions about expected behavior, failure modes, or success criteria, which by definition imports or elicits the very external knowledge the claim disclaims. If interrogation supplies that knowledge, the reported gains in coverage, stage span, redundancy, and Spearman validity (ρ=0.72 on scientific QA) cannot be attributed to a pure zero-label method, and the comparison to AutoMetrics and DynamicRubric baselines becomes inconclusive.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Litmus, a zero-label system that designs evaluation metrics for AI pipelines by analyzing source code and performing targeted interrogation to elicit what must be measured and why, then converts those into constraints for per-stage metric portfolios. Evaluated on three code-defined pipelines (financial account grouping, scientific QA, inherent risk assessment) against AutoMetrics and DynamicRubric baselines, Litmus is claimed to achieve broadest or tied-broadest concern coverage, span more stages, near-zero redundancy, and top validity (Spearman ρ=0.72 on scientific QA vs. <0.47 for all baselines; within CI for two audit components).","tokens_in":1973,"tokens_out":566,"duration_ms":19456,"significance":"If the zero-label property and empirical comparisons hold, the work supports a shift from automatic metric implementation to specification, providing a principled way to justify metrics without labels. The use of real pipelines, explicit coverage/redundancy metrics, and label-based validity checks (while designing without labels) are strengths that could influence evaluation practices for agentic systems.","major_comments":[{"comment":"The load-bearing zero-label claim (§3, interrogation protocol description) rests on the assumption that targeted interrogation can identify evaluation intent from code alone without importing external goal knowledge. The protocol must surface expected behaviors and failure modes to produce the reported coverage and ρ=0.72 gains; any effective questions about success criteria appear to elicit the very knowledge the method disclaims. This needs explicit clarification on question generation and knowledge boundaries, as it directly affects whether performance advantages can be attributed to a pure zero-label regime versus the interrogation step.","section":"§3 (targeted interrogation and zero-label definition)"},{"comment":"Table reporting Spearman correlations (scientific QA row): the validity comparison uses per-row quality labels for evaluation, yet the design is zero-label. The paper should detail how the elicited concerns map to the label criteria used for ρ computation; without this, it is unclear whether the 0.72 result reflects alignment with the intended goals or an artifact of the validation labels.","section":"Experimental results table (scientific QA pipeline)"}],"minor_comments":[{"comment":"The abstract states 'three DynamicRubric baselines' but does not name or differentiate them; the methods section should list their exact configurations for reproducibility.","section":"Abstract and §4"},{"comment":"Notation for 'concern coverage' and 'stage span' is introduced without a formal definition or pseudocode; adding a small table or equation would improve clarity.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the insightful comments, which help strengthen the presentation of the zero-label property and validation procedure. We address each major comment below and will incorporate clarifications in the revised manuscript.","responses":[{"response":"The interrogation protocol relies exclusively on a fixed library of general question templates generated from static code analysis (control-flow graphs, type signatures, and exception paths) with no domain-specific success criteria supplied as input. These templates are identical across all three pipelines. We acknowledge that the current §3 description leaves the knowledge boundary implicit. In revision we will add the precise template-generation algorithm, an example trace for the scientific QA pipeline, and an explicit statement that no external goal statements or label-derived knowledge enter the process. This will allow readers to verify that the reported gains derive from systematic code-driven elicitation rather than imported knowledge.","revision_made":"yes","referee_comment":"[§3 (targeted interrogation and zero-label definition)] The load-bearing zero-label claim (§3, interrogation protocol description) rests on the assumption that targeted interrogation can identify evaluation intent from code alone without importing external goal knowledge. The protocol must surface expected behaviors and failure modes to produce the reported coverage and ρ=0.72 gains; any effective questions about success criteria appear to elicit the very knowledge the method disclaims. This needs explicit clarification on question generation and knowledge boundaries, as it directly affects whether performance advantages can be attributed to a pure zero-label regime versus the interrogation step."},{"response":"The quality labels serve only as an external validation instrument and were withheld during concern elicitation and metric construction. Each Litmus-elicited concern is mapped post hoc to the label rubric dimensions (factual correctness, reasoning completeness, source attribution) by matching the natural-language description of the concern to the rubric item it most directly addresses. We will insert a new subsection (and accompanying table) that lists the 14 elicited concerns for the scientific QA pipeline alongside their corresponding label dimensions, together with the Spearman contribution of each mapped group. This addition will make transparent that the ρ=0.72 improvement tracks better coverage of the intended evaluation dimensions rather than an artifact of the validation labels.","revision_made":"yes","referee_comment":"[Experimental results table (scientific QA pipeline)] Table reporting Spearman correlations (scientific QA row): the validity comparison uses per-row quality labels for evaluation, yet the design is zero-label. The paper should detail how the elicited concerns map to the label criteria used for ρ computation; without this, it is unclear whether the 0.72 result reflects alignment with the intended goals or an artifact of the validation labels."}],"tokens_in":1527,"tokens_out":561,"duration_ms":15649,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Hey, the Litmus paper tries to automate metric specification for AI pipelines by pulling intent out of source code plus targeted interrogation, all without labels. The new angle is treating specification as the hard part rather than just implementation, and they show it on three real pipelines: financial grouping, scientific QA, and risk assessment.\n\nWhat works is the empirical side. Litmus gets the broadest concern coverage, hits more stages, keeps redundancy near zero, and posts the highest Spearman correlation with per-row quality labels on every pipeline, with a clear win on scientific QA at 0.72 versus under 0.47 for the baselines. The portfolio construction step looks reproducible from the description.\n\nThe soft spot is exactly the one the stress-test flags. The method is sold as zero-label, yet the interrogation step has to ask about expected behavior and failure modes to be useful. That step necessarily surfaces or imports knowledge of the intended goals, so the reported gains in coverage and validity can't be attributed to a pure zero-label process. The comparison to AutoMetrics and DynamicRubric baselines therefore sits on shaky ground. No other major flaws jump out; the experiments are straightforward and the citation pattern is normal for the area.\n\nThis is for people who build or audit evaluation systems for deployed agents. A reader working on metric design or pipeline monitoring would pick up usable ideas from the examples even if they disagree with the framing. It deserves a serious referee to press on whether the interrogation protocol can be made genuinely zero-label or whether the claim needs tightening.","headline":"Litmus's zero-label framing doesn't hold up because targeted interrogation imports goal knowledge, but the coverage and validity results on the three pipelines are worth referee time.","tokens_in":2464,"tokens_out":385,"would_cite":false,"duration_ms":14398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Litmus derives evaluation metrics for AI systems directly from their source code without using any labels.","keywords":["zero-label","metric specification","AI evaluation","code-driven","pipeline metrics","evaluation intent","metric portfolio"],"falsifier":"An experiment where experts identify a key evaluation concern from the pipeline's purpose that Litmus's code analysis and questions entirely miss, resulting in incomplete metric coverage.","tokens_in":2748,"feed_emoji":"🔍","tokens_out":425,"duration_ms":13974,"temperature":0.7,"pith_summary":"Litmus is a system that extracts evaluation goals by examining the source code of AI pipelines and asking targeted questions about them. It then uses those goals to build a portfolio of metrics for each stage of the pipeline. The paper shows this method covers more concerns, spans more stages, and has less redundant metrics than existing approaches. It also achieves higher validity when checked against quality labels on three different pipelines, including a strong correlation on scientific question answering. This approach matters because many AI systems have implicit goals that are hard to label in advance.","feed_headline":"Code analysis alone designs valid AI evaluation metrics","feed_subtitle":"Litmus extracts what to measure from pipeline source code and builds portfolios that outperform label-dependent baselines on validity.","key_machinery":"Eliciting evaluation intent from source code and targeted interrogation to generate constraints that construct a per-stage metric portfolio.","core_discovery":"Litmus identifies what must be measured and why from source code and interrogation, then converts those answers into constraints for a justified per-stage metric portfolio. Evaluated on financial account grouping, scientific QA, and inherent risk assessment pipelines, it achieves broadest concern coverage, spans more stages, near-zero redundancy, and first in validity, with Spearman ρ=0.72 on scientific QA compared to less than 0.47 for baselines.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Code analysis creates zero-label AI metrics","Litmus builds metrics from code interrogation","Zero-label system designs valid AI eval portfolios","Source code yields justified per-stage metrics"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Source code analysis combined with targeted interrogation can accurately determine the necessary evaluation criteria without any labels or external goal knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Code analysis creates zero-label AI metrics","Litmus builds metrics from code interrogation","Zero-label system designs valid AI eval portfolios","Source code yields justified per-stage metrics"]},"model":"grok-4.3","cost_usd":0.00486,"raw_usage":{"total_tokens":2341,"prompt_tokens":740,"num_sources_used":0,"completion_tokens":50,"cost_in_usd_ticks":48603000,"prompt_tokens_details":{"text_tokens":740,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1551,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":740,"tokens_out":50,"duration_ms":12465,"temperature":1.0,"reasoning_tokens":1551,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T08:15:01.863387+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment where experts identify a key evaluation concern from the pipeline's purpose that Litmus's code analysis and questions entirely miss, resulting in incomplete metric coverage.","supporting_citations":[],"review_version":1}