{"id":"26159581-900f-45c4-8a99-70d97cb7efa9","arxiv_id":"2505.12792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid local-plus-cloud LLM pipeline cuts input tokens by about 6x and improves F1 on Schwartz value identification benchmarks compared with direct LLM prompting.","lead":"This paper proposes EAVIT, a two-stage pipeline that uses a small local language model to pre-screen which human values a text expresses, then asks a large online LLM to decide only the uncertain cases. The approach cuts prompt length by roughly sixfold while improving accuracy on public value-identification benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Threshold selection on the test set (Table 5, p_low=0.2, p_high=0.8) risks inflating both the F1 gain and the token-reduction claim; validation-based threshold selection is needed to confirm the SOTA margin.","rationale":"After reading the paper, the central claim is that EAVIT obtains SOTA F1 with a 5–8x token reduction. Both halves of that claim are mediated by the candidate-generation band in Section 4.2. The appendix shows the band was tuned on the test set, which is the clearest threat to generalization. I agree with the reader's weakest_assumption; this is the single most load-bearing flaw. I do not see a separate internal inconsistency in the pipeline: the ablation study in Section 5.3 supports the value of explanation-based fine-tuning and generated data, and the output-consistency analysis in Section 5.4 is a positive sign for stability. The virtual-individual study has a generation-evaluation loop that can inflate agreement, but it is illustrative and not needed for the headline claim. The absence of released code and data is an additional limitation, but the proposed concrete test would resolve the main correctness risk without requiring full reimplementation. Because the concern does not by itself falsify the method—a validation-based threshold selection may reproduce similar numbers—I retain the existing CONDITIONAL verdict.","tokens_in":25335,"tokens_out":7145,"duration_ms":70664,"concrete_test":"Have the authors re-run the Touché23-ValueEval and Webis-ArgValues-22 experiments with p_low and p_high selected on the validation split (or by cross-validation on the training split), then report the test F1 and per-sample LLM tokens for the fixed thresholds, including the selected threshold values. If the test F1 drops materially below the reported 0.69 on Touché (or the candidate-set size rises above the implied 0.45k tokens), the claimed SOTA margin is partly an artifact of test-set threshold selection. A useful secondary check is a reliability diagram of the detector's 5-sample probabilities to verify that 'high' and 'low' probability bins correspond to high and low precision on held-out data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core mechanism of EAVIT is the banding rule in Section 4.2: values with average detector probability above p_high are accepted as positives, values below p_low are accepted as negatives, and only the middle band is sent to the LLM. The two quantitative pillars of the paper—the F1 gain over direct prompting (0.69 vs 0.56–0.58 on Touché test, Table 1) and the 0.45k-token cost—both depend on this band being correct. Appendix 8.2 (Table 5) shows that p_low=0.2 and p_high=0.8 were chosen by comparing four threshold pairs on the ValueEval test set and taking the pair with the best F1; no validation split or cross-validation is used. Because L=5 samples are averaged, the probability grid is coarse (multiples of 0.2), so the search is effectively over a small, high-variance grid. The reported 0.69±0.02 does not include the selection step, so the headline margin can be optimistically biased. If the optimal band is dataset-specific, the method's accuracy and token savings on new value systems could both degrade: values outside the band are decided without LLM review, and the candidate-set size directly sets the LLM context length. The paper's own statement that the current setting is 'the optimal setting' after test-set evaluation confirms the selection, so the headline margin should not be taken at face value until thresholds are chosen on development data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EAVIT, a two-stage pipeline for multi-label human value identification. A local Llama2-13b-chat model, fine-tuned with QLoRA using explanation-based prompts and augmented with GPT-4o-mini generated data, serves as a value detector: it is sampled L=5 times per input, producing a per-value relevance probability. Thresholds plow and phigh split the value set into accepted positives (p>phigh), accepted negatives (p<plow), and a candidate set (plow≤p≤phigh) whose definitions are sent to an online LLM (GPT-4o or GPT-4o-mini) for final decisions. Experiments on Touché23-ValueEval and Webis-ArgValues-22 report state-of-the-art F1 (up to 0.69±0.02 on the Touché test set versus 0.56 for the SemEval-2023 best system and 0.58 for GPT-4o sequential CoT prompting) with an LLM token cost of 0.45k per sample versus 2.4k–3.6k for direct prompting. The paper also reports results on ValueNet and a case study that compares value identification from simulated social-media text with psychological questionnaire scores. The authors attribute the gains to the detector's filtering effect, explanation-based fine-tuning, and targeted data generation.","tokens_in":25683,"tokens_out":10782,"duration_ms":95859,"significance":"The core idea of using a cheap local model to filter candidate values and to shorten the definition context for a black-box LLM is simple, practical, and likely to be useful beyond this specific task. The paper includes several strengths: EAVIT's own results are averaged over three runs with standard deviations; the ablation study (Figure 6) isolates the contribution of explanation-based fine-tuning and data generation; the token-cost evaluation (Table 4) is a useful addition; and the virtual-individual case study, while exploratory, connects the method to a psychological validation setting. If the results survive a proper evaluation, EAVIT would provide a cost-effective alternative to direct LLM prompting for value identification, with a five- to eight-fold token reduction and a simultaneous accuracy improvement. However, the headline claim is currently weakened by the fact that the banding thresholds were selected on the test set, which biases both the F1 margin and the token savings; and the baselines lack error bars, so the comparisons are not fully quantified.","major_comments":[{"comment":"The thresholds plow=0.2 and phigh=0.8 are selected by comparing four threshold pairs on the Touché23-ValueEval test set and taking the pair with the highest F1. Since the banding rule determines both the accuracy gain (Table 1: 0.66–0.69 vs 0.56–0.58) and the token reduction (0.45k vs 2.4–3.6k), the headline results are selected optima rather than unbiased estimates for new data. Please re-select thresholds on a validation split or with cross-validation, and report the test-set performance with thresholds fixed before any test-set access, including the variability of the selected thresholds across runs.","section":"Appendix 8.2, Table 5; Section 4.2"},{"comment":"Only the EAVIT rows in Tables 1 and 2 carry standard deviations (over three runs); all baseline numbers are point estimates. The claim that EAVIT 'consistently outperforms' prior methods is thus not fully quantified, and at least one comparison is a tie (ValueNet test accuracy 0.78 for both EAVIT and GPT-4o-mini sequential prompting, Table 2). Please report error bars for all methods, or a paired significance test for the main comparisons (e.g., bootstrap over test items or McNemar's test) so the reader can judge whether the margins are statistically meaningful.","section":"Tables 1 and 2; Section 5.1"},{"comment":"The threshold ablation in Table 5 considers only four threshold pairs on a single test set, and no sensitivity analysis is provided for the other datasets or for the claim that the method transfers to 'any completely defined value system' (Section 8.3). Please report the same threshold grid (or a robustness curve around the chosen pair) on Webis-ArgValues-22 and ValueNet, and discuss whether the optimal thresholds are dataset-dependent, since the validity of the banding rule is the main load-bearing assumption of the pipeline.","section":"Section 4.2 and Appendix 8.2"}],"minor_comments":[{"comment":"'reduces the number of input tokens by up to 1/6' is ambiguous and inconsistent with Table 1, where the reductions are to about 1/5–1/8 of the direct-prompting token count. Please rephrase to 'down to roughly 1/5–1/8' or give the exact factors.","section":"Abstract and Section 1"},{"comment":"'strong non-falsifiability' is not a sensible advantage; the authors likely mean unobtrusiveness or low social-desirability bias. Please reword.","section":"Section 5.2"},{"comment":"There are several typographical errors, including 'GPT-4o-miniGPT-4o-mini' and garbled Unicode fragments (e.g., in the discussion of Figure 3); the manuscript needs a full proofreading pass.","section":"Section 4.1 and throughout"},{"comment":"Table 2 reports only Accuracy on ValueNet, while Table 1 uses F1 (the official metric for this multi-label task); please report F1 or another multi-label metric on ValueNet as well.","section":"Table 2"},{"comment":"The sentence 'We report the average and std of 3 random individual runs' follows the EAVIT setup; please specify explicitly whether this statement applies to all rows of Table 1 or only to EAVIT, and state the number of runs for the baselines.","section":"Section 5.1"},{"comment":"The average candidate set size (3.3 for Touché23-ValueEval) should be accompanied by its distribution, since the token cost (and thus the efficiency claim) depends directly on this quantity.","section":"Section 4.3"},{"comment":"The citation for GPT-2 points to Brown et al. (the GPT-3 paper); please cite the GPT-2 paper (Radford et al.) or clarify the intended model.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising and practical core idea, but the test-set threshold selection is a serious methodological flaw that invalidates the headline numbers as unbiased estimates. The authors should be asked to re-run with validation-based selection and to add error bars to baselines. The manuscript also needs a careful proofreading pass and a clearer description of the 'Value Definition Reflection' step. In my view, the paper is within the scope of the journal and the issues are fixable, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: EAVIT is a two-stage pipeline — a fine-tuned Llama2-13b-chat value detector proposes candidate values, and GPT-4o/GPT-4o-mini makes the final call on just those candidates. The reported gains are real-looking: 0.69 F1 on Touché23-ValueEval vs 0.56–0.58 for strong baselines, and roughly 0.45k LLM tokens per sample instead of 2.4k–3.6k. That combination is worth paying attention to.\n\nWhat is new: the specific packaging. The two-stage local detector + online LLM arbitration is a known pattern in LLM routing, and the training tricks (explanation fine-tuning, targeted data generation) are borrowed from earlier work. But the application to Schwartz value identification is original, and explanation-based fine-tuning for value detectors seems to genuinely help. The paper also does more than the bare minimum: ablations of the threshold grid, API cost numbers, output variance analysis, and a case study with virtual individuals. Good effort on reproducibility for the components.\n\nSoft spots, in proportion. The main one is the threshold selection. Section 4.2 defines the two quantitative pillars — the F1 gain and the token count — on the band between p_low and p_high. Appendix 8.2 (Table 5) reveals that p_low=0.2 and p_high=0.8 were chosen by comparing four threshold pairs on the ValueEval test set and picking the best F1. That is test-set selection. The headline 0.69±0.02 does not include that selection step, so part of the margin may be optimistic. More importantly, token savings depend on the same band, so both claims need validation-based selection before believing the exact numbers. This is fixable: choose thresholds on a dev split and report the resulting test F1.\n\nSecondary issues: baselines have no error bars; the sampling formula in Section 4.2 has a denominator typo (divides by T, presumably should be L); the same GPT-4o-mini family generates training explanations and produces final labels; and the virtual-individual experiment has a generation-then-evaluate loop that can inflate agreement — illustrative, but should be labeled as such.\n\nThe citation pattern is honest and covers the relevant prior work. There is no invented theory or circular reasoning in the evaluation: labels are human-annotated.\n\nWho this is for: anyone working on computational value identification or argument mining who wants a cheaper alternative to direct LLM prompting. It deserves a serious referee — the core idea is sound and the benchmarks are public. I would send it to review with a demand for threshold selection on development data, confidence intervals on baselines, and the typo fixed. The exact SOTA margins are uncertain, but the direction is likely right.","headline":"Plausible, useful two-stage value-identification pipeline with strong reported F1 and token savings, but test-set threshold selection means the headline margins are not yet proven.","tokens_in":26173,"tokens_out":2391,"would_cite":true,"duration_ms":22909,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human value identification from text becomes both cheaper and more accurate when a local model pre-selects candidate values before the LLM arbitrates the remaining candidates.","keywords":["human value identification","large language models","multi-label classification","Schwartz theory of basic values","value detector","candidate set generation","explanation-based fine-tuning","LLM prompting efficiency"],"falsifier":"Run the fixed EAVIT thresholds $p_{low}=0.2$ and $p_{high}=0.8$ on a fresh, out-of-distribution corpus with human-annotated Schwartz values and compare candidate-set recall and final F1 against direct GPT-4o sequential CoT prompting; if true positive values below $p_{low}$ are common enough to leave EAVIT behind the direct baseline, the accuracy half of the central claim is refuted.","tokens_in":12688,"feed_emoji":"⚖️","tokens_out":19870,"duration_ms":266493,"temperature":0.7,"pith_summary":"This paper sets out to show that identifying human values in text, a multi-label classification against a long, definition-heavy taxonomy, is best done by splitting the work between a small local model and a large online one. The local model proposes a short list of candidate values, and the LLM decides only within that list, so the prompt stays short. On the Touché23-ValueEval test set the full EAVIT pipeline reaches F1 0.69, compared with 0.56 for the best SemEval-2023 system and 0.58 for GPT-4o with sequential chain-of-thought prompting, while using 0.45k tokens per sample instead of 2.4k to 3.6k. The paper argues that both the accuracy gain and the cost reduction follow from replacing the full value-system definition with concise candidate-set prompts, and it attributes the detector's usefulness to explanation-based fine-tuning, definition reflection, and LLM-generated training data. A virtual-individual experiment suggests the same approach can recover a person's values from ordinary text more reliably than direct LLM prompting.","feed_headline":"Local prefilter beats full-context GPT-4o on value-tag F1","feed_subtitle":"EAVIT uses a local detector to shrink prompts to ~0.45k tokens and push Touché23 F1 to 0.69.","key_machinery":"The mechanism that carries the argument is the candidate-value-set gate. A local value detector, Llama2-13b-chat fine-tuned with QLoRA and explanation-based supervision, is sampled $L=5$ times per input, producing for each value $V_i$ a mean sampled vote. The pipeline then splits the value set into three zones: values whose vote share is above $p_{high}$ are accepted, values whose vote share is below $p_{low}$ are discarded, and the middle band between $p_{low}$ and $p_{high}$ enters the candidate set $S(T)$ that the LLM arbitrates. The prompt sent to the LLM contains only the definitions of the values in $S(T)$, which averages 3.3 per example, so the context shrinks from the full 2.5k-token value system to roughly 0.45k tokens. This gate is where both savings arise: it removes the long-context degradation the paper attributes to longer prompts, and it cuts API cost by a large factor. Everything else, including data augmentation, definition reflection, and explanation-based fine-tuning, exists to make the gate's probabilities reliable enough that the cheap local model can be trusted outside the band.","core_discovery":"The paper's central claim is that a short prompt built from a candidate set can beat a long prompt built from the full value system for this task, and that the local detector is what makes the short prompt sufficient. EAVIT samples the value detector five times per input, computes the mean sampled vote for each value $V_i$, and uses thresholds $p_{low}=0.2$ and $p_{high}=0.8$; values above the upper threshold are accepted, values below the lower threshold are discarded, and the rest form the candidate set $S(T)$ that GPT-4o or GPT-4o-mini arbitrates using only the definitions of the values in $S$. On the public benchmarks this yields the reported F1 gains, $0.69 \\pm 0.02$ on the Touché23-ValueEval test set versus 0.56 for the SemEval-2023 best system and 0.58 for GPT-4o sequential CoT prompting, while the number of input tokens per sample drops from 2.4k to 3.6k down to 0.45k, about one-fifth to one-eighth of the original. The paper also claims that the candidate gate is stable: the final LLM step suppresses the detector's sampling variance, and the accuracy gains persist on Webis-ArgValues-22 and the ValueNet-augmented dataset.","pith_inferences":["The reported thresholds were selected by maximizing F1 on the test set, so the headline numbers are likely optimistic; a stronger evaluation would tune $p_{low}$ and $p_{high}$ on a separate development split and report the untouched-test numbers.","A testable extension is to replace fixed thresholds with a calibrated uncertainty measure such as vote entropy or margin over the five samples, which could make the gate adaptive to new corpora without re-tuning.","The same prefilter-plus-arbiter pattern should transfer to other definition-heavy multi-label tasks, including moral foundations tagging, legal code classification, and scientific claim typing, wherever the label definitions are long relative to the text.","The passive-value-measurement idea is demonstrated only with GPT-simulated individuals; the decisive experiment would pair real people's social media or interview text with their questionnaire answers on the same individuals."],"forward_implications":["Value tagging at web scale becomes financially feasible because the API token budget is roughly one-fifth to one-eighth of direct prompting, so millions of argumentative texts can be screened for Schwartz values at a fraction of the previous cost.","The method is not tied to the Schwartz taxonomy: any completely specified value system with textual definitions and labeled examples can drive the same detector-plus-arbiter split, per the paper's stated generalization.","For LLM alignment, cheap and accurate value identification makes it practical to audit model outputs for which human values they invoke or oppose, rather than relying on expensive full-context annotations.","For psychology, the virtual-individual case study indicates that passively generated texts can stand in for active questionnaires, with EAVIT predicting the questionnaire-derived values more accurately than the LLM baselines it is compared with.","Because the final LLM step collapses the detector's output variance, the pipeline is suitable for repeated large-scale measurement where output stability matters."],"supporting_citations":[{"why":"Supplies the Schwartz Theory of Basic Values, the taxonomy whose definitions are the long context that EAVIT compresses.","marker":"[Schwartz, 2012]"},{"why":"Defines the Touché23-ValueEval benchmark and the SemEval-2023 task setting that EAVIT compares against.","marker":"[Kiesel et al., 2023]"},{"why":"Documents the long-context degradation that motivates keeping the LLM prompt short.","marker":"[Liu et al., 2023]"},{"why":"Self-Instruct-style prompting used to generate additional training data for the value detector.","marker":"[Wang et al., 2022]"},{"why":"QLoRA enables efficient fine-tuning of the Llama2-13b value detector on consumer GPUs.","marker":"[Dettmers et al., 2023]"},{"why":"The SemEval-2023 Task 4 best system at 0.56 F1, which EAVIT reports beating.","marker":"[Schroter et al., 2023]"},{"why":"Chain-of-thought prompting defines the strongest direct-prompting baseline at 0.58 F1, which EAVIT reports beating.","marker":"[Wei et al., 2023]"},{"why":"Procedure for simulating virtual individuals from questionnaire responses, used in the case study.","marker":"[Aher et al., 2023]"}],"fun_headline_variants":["Value tags: local prefilter cuts tokens 80%, beats GPT-4o F1","1/5 the tokens, better F1: EAVIT's local-first value detection","EAVIT: small model gates value prompts, GPT-4o arbitrates, F1 0.69","Short prompts, local detector, beats full-context LLMs on value ID"],"cache_read_input_tokens":28288,"weakest_assumption_plain":"The pipeline works only if the local detector's five-sample vote can be trusted to say that values scoring under $p_{low}=0.2$ are irrelevant and values scoring over $p_{high}=0.8$ are relevant, without the LLM ever checking them, and those threshold values were themselves chosen to maximize F1 on the test set.","fun_headline_variants_meta":{"raw":{"variants":["Value tags: local prefilter cuts tokens 80%, beats GPT-4o F1","1/5 the tokens, better F1: EAVIT's local-first value detection","EAVIT: small model gates value prompts, GPT-4o arbitrates, F1 0.69","Short prompts, local detector, beats full-context LLMs on value ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3072,"prompt_tokens":1062,"completion_tokens":2010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":1913}},"tokens_in":678,"tokens_out":2010,"duration_ms":15746,"temperature":1.0,"reasoning_tokens":1913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:40.504384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fixed EAVIT thresholds $p_{low}=0.2$ and $p_{high}=0.8$ on a fresh, out-of-distribution corpus with human-annotated Schwartz values and compare candidate-set recall and final F1 against direct GPT-4o sequential CoT prompting; if true positive values below $p_{low}$ are common enough to leave EAVIT behind the direct baseline, the accuracy half of the central claim is refuted.","supporting_citations":[{"cited_title":"An overview of the schwartz theory of basic values","cited_arxiv_id":null,"evidence_quote":"Supplies the Schwartz Theory of Basic Values, the taxonomy whose definitions are the long context that EAVIT compresses."},{"cited_title":"Semeval- 2023 task 4: Valueeval: Identification of human values be- hind arguments","cited_arxiv_id":null,"evidence_quote":"Defines the Touché23-ValueEval benchmark and the SemEval-2023 task setting that EAVIT compares against."},{"cited_title":"Qlora: Efficient fine- tuning of quantized llms,","cited_arxiv_id":null,"evidence_quote":"QLoRA enables efficient fine-tuning of the Llama2-13b value detector on consumer GPUs."},{"cited_title":"Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models","cited_arxiv_id":"2305.08625","evidence_quote":"The SemEval-2023 Task 4 best system at 0.56 F1, which EAVIT reports beating."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting defines the strongest direct-prompting baseline at 0.58 F1, which EAVIT reports beating."},{"cited_title":"Using large language models to sim- ulate multiple humans and replicate human subject studies","cited_arxiv_id":null,"evidence_quote":"Procedure for simulating virtual individuals from questionnaire responses, used in the case study."}],"review_version":1}