{"id":"875fa5ca-95e6-4b04-a67b-626e2b293882","arxiv_id":"2607.17548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Interactive feedback was associated with a more negative perceived-accuracy trend in an objective face-detection task, but no such bias appeared in two subjective text-classification studies.","lead":"Three controlled studies test how letting users give feedback to an AI changes their trust and accuracy estimates. In an objective image-detection task interactive feedback coincided with a more negative perceived-accuracy trend, while in subjective text-classification tasks no negative bias appeared.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Subjectivity is never manipulated or measured; cross-experiment differences are confounded with explanation format, domain, and population, so the central causal moderator claim is not yet supported.","rationale":"The reader's weakest-assumption analysis identified exactly the same load-bearing concern: the paper infers a causal role for subjectivity from a comparison of separate experiments that differ in domain, explanation format, and participant population, without ever manipulating or measuring perceived subjectivity. The authors' own limitation statement in §6.4 confirms this. My reading of the full text adds that the Experiment 1 statistics do not cleanly support the abstract's claim of lowered trust and perceived accuracy: the trust ANOVA is non-significant, and the perceived-accuracy feedback effect sits at the conventional boundary of p = 0.050 and is reported as non-significant. The significant result is a retrospective perception-of-change measure, not the direct accuracy or trust ratings. However, this does not undermine the value of the paper as a conditional contribution with two null-result replications in a subjective context and an honest limitations discussion. The reader's CONDITIONAL verdict is therefore appropriate; no movement is needed, but the abstract should be revised to match the actual inferential support.","tokens_in":23855,"tokens_out":2617,"duration_ms":26352,"concrete_test":"Run a single experiment using the same text-classification interface and explanation format (word highlights) for all participants, manipulating only the perceived subjectivity of the feedback: one condition frames the highlighted-word selections as having objectively correct answers and tells participants they are correcting system errors; the other condition frames the same selections as subjective preferences and tells participants they are tuning the system to their own mental model. Keep all other elements identical across conditions (system accuracy, sample ordering, number of rounds, participant pool). Measure perceived accuracy after each round and trust at the end. If the two conditions diverge in the predicted direction (lower perceived accuracy/trust in the objective-framing condition), the subjectivity account is supported independent of domain and explanation format; if not,","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that task subjectivity causally moderates the effect of HITL feedback on trust and perceived accuracy. This requires that the only systematic difference between the negative-bias experiments (Experiment 1) and the no-bias experiments (Experiments 2 and 3) be the subjectivity of the feedback. But the studies differ in multiple confounded dimensions: feedback modality and explanation role (bounding-box correction vs. word-highlight editing), task domain (image detection vs. text classification), number of trials per round (30 vs. 15), stimulus ordering (fixed vs. counterbalanced), and participant population (MTurk vs. UF students). The authors explicitly acknowledge in §6.4 that \"subjectivity was not a variable that was directly controlled for\" and that the text case \"has more of an element of explainability than the more objective image case.\" Moreover, within Experiment 1, the headline claim is not directly supported by the reported statistics: the trust ANOVA was non-significant, the perceived-accuracy effect of feedback type was p = 0.050 and reported as non-significant, and the only significant feedback-type effect was on the retrospective perception-of-change measure. Thus the abstract's assertion that HITL feedback \"lowered both participants' trust in the system and their perception of system accuracy\" goes beyond what Experiment 1 alone demonstrates, and the cross-experiment comparison cannot separate subjectivity from the other domain/interface differences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports three between-subjects experiments on how providing human-in-the-loop (HITL) feedback affects trust and perceived accuracy. Experiment 1 (n=107) uses a simulated object-detection task with binary vs. interactive feedback and an update-belief manipulation; Experiments 2 and 3 (n=144 and n=94) use a text-classification task with no, decision-based, or explanation-based feedback. The paper's central claim is that in objective-feedback contexts HITL feedback lowers trust and perceived accuracy, while in subjective-feedback contexts no such negative bias occurs, and that the two contexts lead to different perceived accuracy trajectories over time.","tokens_in":24074,"tokens_out":5951,"duration_ms":53198,"significance":"If the central claim were fully supported, the paper would provide a useful design guideline for interactive ML: framing feedback as objective error correction can bias users negatively, whereas subjective feedback may preserve or improve perceptions. The studies have genuine strengths: the simulated systems hold true accuracy constant, avoiding the common confound of improved system performance; Experiment 3 provides a replication with a different population and counterbalanced ordering; effect sizes are reported; and the authors are transparent about the subjectivity limitation in §6.4. However, the headline causal claim is not established by the reported statistics and design, for reasons detailed below. The empirical pattern is interesting and worth reporting, but the paper currently overstates what can be concluded.","major_comments":[{"comment":"The abstract states that in a context with an objectively correct answer, HITL feedback 'lowered both participants' trust in the system and their perception of system accuracy.' Experiment 1 does not support this. For perceived accuracy, the feedback-type effect was F(1,103)=3.71, p=0.050, which the authors themselves report as non-significant; for trust, the two-way ANOVA found no significant effects of feedback type or feedback usage. The only significant feedback-type effect was the retrospective perception-of-change measure (§3.2.2), which is not the same as lower perceived accuracy or lower trust. Additionally, the abstract's 'regardless of whether the system accuracy improved in response to their feedback' is not tested in this paper: system accuracy was constant in all conditions and participants were deceived about updates. The abstract and the conclusions in §7 should be reworde","section":"§3.2.1, §3.2.3, Abstract"},{"comment":"The central claim that task subjectivity causally moderates the effect of HITL feedback is based on comparing separate studies that differ on many dimensions. Experiment 1 used bounding-box correction on images; Experiments 2 and 3 used highlighted-word editing in text. The studies also differ in trials per round (30 vs. 15), stimulus ordering (fixed vs. counterbalanced), and participant population (MTurk vs. University of Florida students), and only the text condition includes a visible explanation of the system's reasoning. The authors acknowledge in §6.4 that 'subjectivity was not a variable that was directly controlled for' and that the text case 'has more of an element of explainability.' Without a direct manipulation of perceived subjectivity within a constant paradigm, or at minimum a measurement of perceived subjectivity, the observed cross-experiment differences cannot be attrib","section":"§6.4, §4.2.1, §5.1"},{"comment":"The conclusion that 'no such negative bias was observed' in the subjective contexts rests on non-significant main effects of feedback condition on perceived accuracy (Experiment 2: F(2,137)=0.951, p=0.389; Experiment 3: F(2,91)=0.871, p=0.422). A non-significant p-value is not evidence for the absence of an effect unless accompanied by equivalence testing or a Bayes-factor analysis. Because the paper's central comparison is between a significant effect in one study and null effects in two others, the asymmetry should be quantified rather than inferred from p-values alone.","section":"§4.3.1, §5.3.1"}],"minor_comments":[{"comment":"The text says 'As in Experiment 3, this decision to deceive participants...' but Experiment 3 is introduced later; this should likely refer to Experiment 2 or Experiment 1.","section":"§4.2.1"},{"comment":"Typo: 'without into their systems' should read 'without negatively biasing their users' or similar.","section":"§7"},{"comment":"The use of 'distrust' versus 'mistrust' is confusing. In the subjective context participants perceived the system as improving over time; describing this as 'mistrust' seems inconsistent with the usual meaning of mistrust as insufficient trust. Clarify the intended distinction.","section":"Abstract, §7"},{"comment":"In the provided manuscript version, Figure 3 appears to contain duplicate panels. Please verify the final figure.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is competent and the studies are carefully controlled in several respects. The main problem is that the title, abstract, and conclusions make a stronger causal claim than the design and statistics support. If the authors reframe the contribution as an exploratory cross-context comparison and add a direct manipulation or measurement of perceived subjectivity, the paper could be suitable for publication. In its current form, the central claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a careful, honestly reported extension of the authors' earlier HITL feedback study, with two new text-classification experiments and a reworked version of the original image-detection experiment. What it does not yet establish is the causal claim in the title and abstract — that task subjectivity is what drove the different trust effects. Subjectivity is never manipulated or measured; the objective and subjective studies differ in task domain, explanation format, trial count, stimulus ordering, and participant pool.\n\nWhat's genuinely new: Experiments 2 and 3 provide fresh data in a subjective text-classification context where, unlike the image task, no negative bias from interactive feedback appears. Experiment 1 adds two conditions (binary-with-update, interactive-without-update) to separate belief that the system updates from the act of providing richer feedback. The paper's reporting is unusually transparent for a preprint — full ANOVAs, effect sizes, exclusion counts, and a limitations section that explicitly concedes subjectivity was not directly controlled and that the text case 'has more of an element of explainability.' That is honest.\n\nWhere it falls short: the abstract's opening claim overstates Experiment 1. In the reported statistics, the trust ANOVA was non-significant, the perceived-accuracy effect of feedback type was p=0.050 and reported as non-significant; the only feedback-type effect that reached significance was the retrospective perception-of-change measure. So the claim that interactive feedback 'lowered both participants' trust and perception of system accuracy' rests on the earlier study [26] rather than on the current experiment. And since the cross-experiment comparison bundles together multiple differences, the 'subjectivity matters' conclusion is plausible but not demonstrated. A direct manipulation of feedback objectivity within a single task, or at least a no-feedback control in E1, would move this from conditional to convincing.\n\nThis paper is for researchers working on trust in AI and interactive ML. It deserves serious peer review — the empirical work is careful and the question is important — but the abstract and interpretation need to be recalibrated to match the evidence, and the moderation hypothesis needs a cleaner test. With those revisions it would be a solid contribution.","headline":"A careful, transparently reported three-experiment extension, but the title and abstract claim that task subjectivity is the moderator goes beyond the evidence; worth reviewing, needs revisions.","tokens_in":24634,"tokens_out":3962,"would_cite":true,"duration_ms":32724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task subjectivity—not the act of giving feedback—decides whether users trust an AI more or less","keywords":["trust in automation","human-in-the-loop","perceived accuracy","feedback subjectivity","interactive machine learning","user feedback","explainable AI","task context"],"falsifier":"Use one task and one interface in which the same feedback action is described to randomly assigned participants either as fixing objectively wrong outputs or as tuning the system to their personal judgment, keeping explanations, examples, and true accuracy identical; if the trust difference appears in both framings or in neither, task subjectivity is not the causal driver.","tokens_in":23645,"feed_emoji":"🤖","tokens_out":8659,"duration_ms":72893,"temperature":0.7,"pith_summary":"This paper claims that task subjectivity determines whether asking users for feedback helps or hurts their trust in an intelligent system. In an object-detection context where feedback corrects objectively wrong outputs, the authors find that interactive feedback leads users to perceive lower accuracy and to trust the system less, and that believing the system updates from their feedback does not offset the effect. In a text-classification context where feedback expresses subjective judgments about which words matter, no such negative bias appears; instead users in all conditions perceive the system as getting more accurate over time even though its true accuracy is constant. The paper argues the objective framing turns feedback into 'fix my mistakes,' while the subjective framing turns it into 'learn my thinking,' and recommends that human-in-the-loop features be framed as user agency rather than error correction.","feed_headline":"User feedback hurts trust only when errors feel objective","feed_subtitle":"Three experiments: the same feedback act lowers trust or inflates it, depending on task subjectivity.","key_machinery":"The central object is task subjectivity—whether the feedback has a single objectively correct answer (a face is or is not in a box) or is open to judgment (which words best justify a topic label). The experiments operationalize it by moving the same feedback interaction from bounding-box correction in images to explanation-word re-ranking in text, while holding the simulated system's true accuracy at 80 percent and telling participants the model was updating when it was not. The mechanism the authors propose to explain their results is error salience: objective errors are obvious and memorable, so correcting them makes the system feel worse than it is; subjective feedback makes users compare","core_discovery":"The paper's central claim is that the meaning users assign to giving feedback—correcting an objectively wrong output versus expressing a subjective judgment—determines whether human-in-the-loop interaction helps or hurts their view of an intelligent system. In an object-detection task where bounding boxes either do or do not contain a face, participants who gave interactive corrections rated the system as less accurate over time and trusted it less than did participants who only gave a yes/no accuracy check; whether they believed the system was updating from their feedback made no difference. In a text-classification task where participants adjusted which highlighted words best explained a t","pith_inferences":["A clean test of the paper's explanation would hold the task and interface constant and randomly frame the identical feedback action as either correcting errors or expressing preference; the paper's own limitation section says subjectivity was not directly controlled for.","If error salience is the underlying driver, then interface choices that reduce how long users dwell on each error—such as batching corrections or showing aggregate accuracy alongside mistakes—might blunt the negative bias in objective tasks.","The pattern suggests a calibration dilemma: making feedback feel subjective may increase satisfaction and perceived accuracy, but if perceived accuracy outruns true accuracy, designers have simply traded distrust for automation bias.","Because all three studies used a simulated, non-updating system and one-session tasks, the findings most directly apply to first impressions; whether the effects persist after users see real model updates or across longer use remains untested."],"forward_implications":["In domains where system errors are obviously wrong to users, adding interactive error-correction feedback can lower perceived accuracy and trust even if the model genuinely improves from the feedback.","Telling users their feedback is being used does not remove the negative bias; the negative effect comes from the act of correcting errors, not from uncertainty about whether updates happen.","In subjective domains, explanation-based feedback can avoid the distrust penalty, but it comes with a different risk: users may come to believe the system is improving over time when its accuracy is flat, which can lead to over-reliance.","Designers who want human-in-the-loop features without biasing user trust should frame feedback as adjusting the system toward the user's judgment, not as correcting objective errors."],"fun_headline_variants":["Corrections that feel objective erode user trust in AI","Feedback helps AI—but only if the task is subjective","Why giving AI feedback backfires on objective tasks","Objective feedback makes users distrust even improving AI","Subjective feedback boosts trust; objective feedback kills it"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the different results came from whether the feedback felt objective or subjective, but the studies never directly controlled or measured that feeling, and the objective and subjective tasks also differed in explanation style, example order, and participant pool—a limitation the paper itself acknowledges.","fun_headline_variants_meta":{"raw":{"variants":["Corrections that feel objective erode user trust in AI","Feedback helps AI—but only if the task is subjective","Why giving AI feedback backfires on objective tasks","Objective feedback makes users distrust even improving AI","Subjective feedback boosts trust; objective feedback kills it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3198,"prompt_tokens":785,"completion_tokens":2413,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2337}},"tokens_in":529,"tokens_out":2413,"duration_ms":14546,"temperature":1.0,"reasoning_tokens":2337,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:39:09.770779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use one task and one interface in which the same feedback action is described to randomly assigned participants either as fixing objectively wrong outputs or as tuning the system to their personal judgment, keeping explanations, examples, and true accuracy identical; if the trust difference appears in both framings or in neither, task subjectivity is not the causal driver.","supporting_citations":[],"review_version":1}