{"id":"83fc0680-a3fe-458c-8f72-11d2601d1aa2","arxiv_id":"2505.09747","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors propose 'healthy distrust' as a justified, careful stance toward AI usage practices, arguing that distrust can coexist with trust and is key to respecting human autonomy.","lead":"This paper argues that distrusting an AI system can be a healthy and justified stance, not a failure, especially when the system is used in ways that conflict with your interests. It introduces the term 'healthy distrust' to help researchers and regulators understand and protect human autonomy around AI.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The normative claim that healthy distrust should be cultivated depends on an underspecified 'healthy' criterion; without a demarcation between justified and pathological distrust, the prescriptive conclusion is not secured.","rationale":"The reader's verdict (CONDITIONAL) is one I agree with, but the weakest-assumption analysis points at the Luhmann risk/danger transfer. I read that as a supporting illustration rather than a load-bearing premise: the final definition in Section 7 does not invoke risk/danger, and the descriptive claim that distrust can be justified by social context is supported by the insurance and racial-bias examples independently. The load-bearing weakness is the normative predicate 'healthy.' The paper draws a line between the dangerous generalized distrust it cites in Section 5 and the healthy distrust it recommends, but the criteria offered in Sections 6 and 7 are too weak to draw that line: intuition, trust in some alternative, and a relation to what the technology does are also present in paradigmatically unhealthy conspiracy-based distrust. Because the paper explicitly includes pre-rational hesitation in the concept, the boundary problem is internal, not imported from a contested sociological framework. A demarcation test with matched vignettes would show whether the concept can bear its normative weight. Therefore I would keep the CONDITIONAL verdict, but the condition should be a principled criterion for 'healthy,' not the acceptance of Luhmann's distinction.","tokens_in":19993,"tokens_out":6929,"duration_ms":76257,"concrete_test":"Apply the Section 6/7 criteria literally to two matched vignettes. Vignette A: a patient distrusts a health-insurance scoring AI because she has experienced discriminatory denials; her distrust is grounded in that experience, she trusts her own judgment, and she can explain what the system does. Vignette B: a user distrusts the same AI because he affectively holds a conspiracy belief that it is a government surveillance tool; he trusts alternative online sources and can also state what the system does. Ask two independent judges to classify each vignette as healthy or unhealthy using only the paper's stated criteria. If Vignette B is classified as healthy, or if the two judges cannot agree because the criteria are indeterminate, the normative concept lacks the needed demarcation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two components: a descriptive one (some affected people have good reason to distrust AI even when the system meets technical trustworthiness criteria) and a normative one (such distrust is healthy and ought to be fostered). The descriptive component is well illustrated by the health-insurance scoring example and by the racial-bias discussion, and it does not depend on Luhmann's risk/danger distinction. The normative component, however, rests on the predicate 'healthy' being able to separate warranted distrust from unwarranted, harmful distrust. Section 6 says healthy distrust must be rooted in 'knowledge, reasoning, or at least intuition' and must trust something that has 'some relation to what the technology does, or is meant to do'; Section 7 defines healthy distrust as including 'pre-rational hesitancy and reluctance.' These criteria are too broad: an affectively held conspiracy belief about an AI system is also rooted in intuition, also trusts alternative sources, and also makes a claim about what the technology does. The paper itself warns in Section 5, citing Thielmann and Hilbig, that generalized distrust is a precondition of conspiracy mentality, but it gives no principle marking the boundary between that dangerous distrust and the distrust it recommends cultivating. The objection is acknowledged in Section 7 ('How long should an institution wait...') but not answered. Until such a demarcation criterion is provided, the prescriptive conclusion that healthy distrust should be fostered is ungrounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that distrust toward AI systems can be justified and desirable, coining the term 'healthy distrust' to describe a partially rational, partially affective careful or negative stance toward specific AI usage practices in specific socio-technical contexts. The authors distinguish a descriptive claim (some affected people have good reason to distrust AI even when the system satisfies technical trustworthiness criteria) from a normative claim (such distrust is healthy, ought to be fostered, and is needed to protect human autonomy). They support this by reviewing notions of trust in history, philosophy, psychology, and computer science; by discussing examples such as health-insurance scoring and racial bias in AI; and by addressing objections in Sections 6 and 7, where they propose a preliminary definition and acknowledge unresolved conceptual overlap.","tokens_in":20237,"tokens_out":6167,"duration_ms":65437,"significance":"If the concept can be made precise, healthy distrust would fill a genuine gap in the trustworthy-AI literature, which largely treats trust as the desired endpoint and distrust as a failure or obstacle. The paper is strongest in its descriptive argument: the health-insurance and racial-bias examples show that distrust can be reasonable even for systems that meet formal trustworthiness requirements, and the review usefully highlights psychological evidence that trust and distrust can co-exist and that distrust has cognitive benefits. The authors also deserve credit for explicitly engaging with power asymmetries, for arguing that distrust is not a property of a system but of a stance toward a usage practice, and for connecting the concept to AI literacy, human oversight, and informed consent. The interdisciplinary literature review is broad and well-referenced, and the central claim is not defined circularly in terms of the authors' own prior work.","major_comments":[{"comment":"The normative component of the central claim is not secured because the paper never supplies a demarcation criterion for 'healthy'. Section 6 defines healthy distrust as rooted in 'knowledge, reasoning, or at least intuition' and as requiring that the distrust trust something with 'some relation to what the technology does, or is meant to do'; Section 7 then includes 'pre-rational hesitancy and reluctance' in the concept. These conditions are also satisfied by a conspiracy belief about an AI system that is held on the basis of an intuitive alternative relation and that makes claims about what the technology is meant to do. Since Section 5 cites Thielmann and Hilbig (2023) as showing that generalized distrust is a causal precondition of conspiracy mentality, the reader has no way to tell whether the recommended cultivation of distrust promotes autonomy or feeds pathology. The objection in Section 7 ('How long should an institution wait...') is acknowledged but not answered with a criterion; the reply that AI hype warrants delay supports a permission for distrust in specific cases, not the general normative conclusion that distrust 'ought to be fostered and cultivated'. A concrete fix would be to add explicit conditions such as proportionality to evidence, openness to revision, targeting of specific usage practices rather than diffuse suspicion, and action-readiness.","section":"Section 6 and Section 7"},{"comment":"The transfer of Luhmann's risk/danger distinction to AI usage is load-bearing for the claim that data subjects cannot meaningfully trust AI, but it is presented as a given rather than a contested theoretical choice. There are two specific gaps. First, Luhmann's distinction links danger to confidence ('Zuversicht'), not to distrust, so the move from 'data subjects face danger rather than risk' to 'data subjects should adopt a stance of healthy distrust' is not licensed by Luhmann himself. Second, the paper applies 'data subjects' as a blanket category, but a data subject who can decline a service, switch providers, or consent to processing does have decision-making scope; in such cases Luhmann's own framework would classify the situation as risk rather than danger, and trust would become meaningful again. The descriptive examples in Sections 2 and 6 (health-insurance scoring, racial bias) do not need this sociological premise. The paper could weaken the claim to 'in contexts where affected people lack decisional control, distrust is a legitimate stance' and thereby avoid making the argument depend on acceptance of the Luhmannian frame. As written, the argument for why distrust rather than confidence is the appropriate stance for data subjects loses its basis for readers who do not accept that framework.","section":"Section 4"},{"comment":"There is an unclosed gap between the hedged conceptual claims and the prescriptive conclusion. Section 7 says healthy distrust 'may be a necessary, or at least helpful, part of such a critical stance' and acknowledges that it may lead to under-utilization of AI; the rebuttal is that past harms and AI hype make delay 'warranted' for many contemporary usage practices. This supports a permission to distrust in cases with identifiable reasons, but it does not support the stronger prescription in Section 1 that distrust 'ought to be fostered and cultivated'. To support the stronger claim, the paper would need to specify which actors (individuals, institutions, educators), which usage practices, and what safeguards against the documented costs of generalized distrust (Thielmann and Hilbig 2023) are intended. Absent such scope conditions, the normative claim is broader than the argument.","section":"Section 1 and Section 7"}],"minor_comments":[{"comment":"The text contains a duplicated article: 'the different perspectives on trust proposed by the the predicative vs. the affective view of trust' should read 'by the predicative vs. the affective view'.","section":"Section 4"},{"comment":"The phrase 'finally make an an informed decision about it' contains a duplicated 'an'.","section":"Section 7"},{"comment":"The acknowledgement line 'WegratefullyacknowledgefundingbytheGermanResearchFoundation' lacks spaces between words; this appears to be a formatting error.","section":"Acknowledgements"},{"comment":"The footnote explaining the choice of 'healthy distrust' says the pairing is with 'the clearly negative word mistrust', while the rest of the paper consistently uses 'distrust'; these terms should be reconciled or the distinction explained.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth reading and worth refereeing, but the prescriptive conclusion is not yet secured. The descriptive contribution is solid: the paper identifies a real phenomenon—people can be justified in distrusting an AI system even when it meets trustworthy-AI checklists—and connects it to autonomy, power, and social embeddedness. It is honest, interdisciplinary, and unusually careful about acknowledging overlap with prior concepts (Mayo's Cartesian mindset, Lewicki's trust/distrust coexistence, Mühlfried, Schul). The health-insurance scoring example and the racial-bias discussion carry the descriptive argument, and they do not depend on the Luhmann risk/danger distinction that the reader flagged. That distinction is a framing choice, not a load-bearing proof, and the authors treat it as given rather than defended.\n\nThe real soft spot is the 'healthy' criterion. The stress-test note gets this right. Section 6 says healthy distrust must be rooted in knowledge, reasoning, or intuition and must trust something with some relation to what the technology does; Section 7 includes pre-rational hesitancy and reluctance. That is too broad: a conspiracy-minded distrust of an AI is also rooted in intuition and trusts alternative sources. The paper cites Thielmann and Hilbig on generalized distrust as a precondition of conspiracy mentality, acknowledges the objection in Section 7 ('How long should an institution wait...'), but does not answer it. So the descriptive claim is well argued; the normative claim that distrust should be cultivated lacks a demarcation principle. That is a significant gap, but not fatal for a conceptual paper whose authors present the definition as preliminary and the term as a starting point.\n\nWhat is genuinely new is the synthesis: the paper assembles history, sociology, psychology, and philosophy into a reusable term for AI ethics and gives distrust a positive role in protecting autonomy. Citation pattern looks fine; self-citations are empirical support, not circularity.\n\nWho is this for? AI ethics researchers, HCI people working on trust and explainability, and anyone doing conceptual work on human oversight. It does not resolve an empirical question, and operationalization remains. But it is a serious, honest contribution that deserves referee time.","headline":"A serious, honest conceptual paper: the descriptive case for justified distrust is solid, but the normative 'healthy' boundary is underspecified and needs work.","tokens_in":20781,"tokens_out":2662,"would_cite":true,"duration_ms":26726,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that justified distrust toward AI usage practices—healthy distrust—is a desirable, cultivable stance rather than a failure of trust.","keywords":["healthy distrust","trustworthy AI","distrust","human autonomy","AI usage practices","AI literacy","human oversight","risk and danger"],"falsifier":"A field study in a high-stakes automated decision context (e.g., hiring or health screening) could test the central claim by measuring users' distrust, felt autonomy, and decision quality: if people with low distrust made better-calibrated decisions and reported more autonomy than people with high distrust, holding system accuracy and social context fixed, the paper's normative case for healthy distrust would be undercut.","tokens_in":19771,"feed_emoji":"🤖","tokens_out":7840,"duration_ms":73821,"temperature":0.7,"pith_summary":"This paper argues that the drive to make AI trustworthy has overlooked a legitimate and desirable state: justified distrust of AI usage practices, which the authors name 'healthy distrust.' They define it as a partly rational, partly affective careful or negative stance toward a particular AI usage practice in a particular socio-technical context—an intuition that something is not right about how the system is being used. Drawing on history, sociology, psychology, and philosophy, they show that existing concepts of trust and distrust either treat distrust as the absence of trust or tie it to detecting deception in human agents, neither of which fits the AI case. The paper claims that healthy distrust is not opposed to trust: it is a precondition for meaningful trust, meaningful human oversight, and informed consent, and it should be cultivated rather than engineered away. A sympathetic reader would take away that whether an AI is trustworthy is a separate question from whether a person does or should trust it.","feed_headline":"Distrust of AI can be healthy, not a bug","feed_subtitle":"A conceptual paper argues that justified distrust of AI usage protects human autonomy and should be fostered, not engineered away.","key_machinery":"The load-bearing conceptual machinery is Luhmann's distinction between risk and danger, transferred to AI. Risk is possible damage resulting from one's own decisions; danger is possible damage from external sources that one cannot avoid by deciding. In AI, the practitioner or deploying institution takes a risk, while the data subject—the person affected by the decision—faces a danger outside their decision-making scope; because Luhmann links trust to risk and confidence to danger, the paper concludes that expecting data subjects to trust an AI is meaningless. The second piece of machinery is the psychological and organizational finding that trust and distrust are separable, coexisting dimensions rather than opposites, which lets the authors frame healthy distrust as an accompanying stance rather than a failure. Together these tools define healthy distrust as a specific, context-bound stance—rational and affective—toward a usage practice, whose 'health' is judged by what it trusts in turn and by the power context that enables it.","core_discovery":"The central claim is that distrust can be justified and normatively appropriate toward AI systems even when the systems satisfy every criterion of trustworthy AI, because trustworthiness is a property of the system while distrust concerns the user's stance toward a usage practice embedded in a social context of power relations and interests. The authors propose 'healthy distrust' for this stance: partially rational and partially affective, a careful or negative attitude or intuition that something is not right about a specific usage practice. They argue that such distrust is often grounded not in technical failure but in the social embedding of the system—an insurer's incentive to overestimate risk, a workplace rule that forces compliance, a history of discriminatory outcomes—and that it can be a way of asserting autonomy under automation. On this account, distrust and trust are not opposite ends of one scale but separable dimensions that can coexist: one can trust a system to do what it was trained for while distrusting its use in one's case. The paper therefore positions healthy distrust as a necessary, or at least helpful, component of a critical stance that respects human autonomy, and as a complement to AI literacy centered on recognizing questionable usage practices.","pith_inferences":["Going beyond the paper: healthy distrust could be operationalized as a measurable construct distinct from trait distrust and generic skepticism, giving empirical AI research an outcome variable beyond trust scales.","Going beyond the paper: if healthy distrust matters, interfaces that preserve friction or a deliberate pause for scrutiny should improve the calibration of user trust, especially in high-stakes decisions—a directly testable design hypothesis.","Going beyond the paper: the account connects to explainable AI evaluation, where explanations should be assessed by whether they let users accurately distrust out-of-scope or misused systems, not only by whether they increase trust.","Going beyond the paper: because healthy distrust depends on resources and alternatives, the power-sensitive reading implies that institutions deploying AI carry a duty to make distrust actionable through training, contestability, and opt-outs."],"forward_implications":["A system can be certified trustworthy and still be justifiably distrusted, so trustworthiness checklists are not enough to license deployment; the social embedding of the usage matters.","Design and regulation should aim to preserve room for healthy distrust—pause, questioning, override—rather than maximize smooth adoption, because that room is what makes trust meaningful.","Human oversight and human-in-the-loop concepts presuppose some distrust: overseers must be able to anticipate failure or suspect bad practice to intervene at all.","Informed consent toward AI usage practices is only meaningful if people can engage critically with the practice; healthy distrust supports that critical engagement.","Fostering healthy distrust belongs in AI literacy education, alongside knowledge and skills for using AI well, as the knowledge and skills for recognizing questionable usage."],"supporting_citations":[{"why":"Supplies the risk-versus-danger distinction that grounds the argument that data subjects face dangers and therefore cannot meaningfully trust AI.","marker":"Luhmann, 1990"},{"why":"Documents the historical valorization of trust and the negative connotation of distrust, which the term 'healthy distrust' is meant to counter.","marker":"Frevert, 2013"},{"why":"Defines distrust as confident negative expectations and makes the case that trust and distrust are separate dimensions that can coexist.","marker":"Lewicki et al., 1998"},{"why":"Provides empirical evidence that a distrust mindset has cognitive value and that trust and distrust are not simply opposites.","marker":"Schul et al., 2008"},{"why":"Offers the Cartesian evaluative mindset as an alternative to both trust and distrust, which the paper uses to sharpen what healthy distrust adds.","marker":"Mayo, 2024"},{"why":"Supplies the point that any distrust must in turn trust something else, used to define what makes distrust 'healthy'.","marker":"Wittgenstein, 1969"},{"why":"Provides the power-sensitive analysis of how technologies enable and disable practices, which the paper uses to situate healthy distrust in contexts of power.","marker":"Matzner, 2024"},{"why":"Empirical survey supporting the claim that an AI's trustworthiness does not imply that users do or should trust the system.","marker":"Visser et al., 2025"}],"fun_headline_variants":["Justified AI distrust is healthy, not a flaw","Healthy distrust: a necessary stance for AI users","AI distrust isn't ignorance; it's autonomy in action","Distrust of AI use can be justified and healthy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the Luhmann-inspired premise that people affected by AI decisions face a danger outside their own decision-making scope, so trust in the AI is not a meaningful option for them; if one rejects that framing or finds contexts where affected people genuinely choose the automation, the conclusion that distrust is the appropriate stance for them weakens.","fun_headline_variants_meta":{"raw":{"variants":["Justified AI distrust is healthy, not a flaw","Healthy distrust: a necessary stance for AI users","AI distrust isn't ignorance; it's autonomy in action","Distrust of AI use can be justified and healthy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2779,"prompt_tokens":911,"completion_tokens":1868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1805}},"tokens_in":527,"tokens_out":1868,"duration_ms":16712,"temperature":1.0,"reasoning_tokens":1805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:24:51.162941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A field study in a high-stakes automated decision context (e.g., hiring or health screening) could test the central claim by measuring users' distrust, felt autonomy, and decision quality: if people with low distrust made better-calibrated decisions and reported more autonomy than people with high distrust, holding system accuracy and social context fixed, the paper's normative case for healthy distrust would be undercut.","supporting_citations":[{"cited_title":"Technology, environment and social risk: a systems perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the risk-versus-danger distinction that grounds the argument that data subjects face dangers and therefore cannot meaningfully trust AI."},{"cited_title":"Vertrauensfragen: Eine Obsession der Moderne","cited_arxiv_id":null,"evidence_quote":"Documents the historical valorization of trust and the negative connotation of distrust, which the term 'healthy distrust' is meant to counter."},{"cited_title":"The value of distrust","cited_arxiv_id":null,"evidence_quote":"Provides empirical evidence that a distrust mindset has cognitive value and that trust and distrust are not simply opposites."},{"cited_title":"Trust or distrust? Neither! The right mindset for confronting disinformation","cited_arxiv_id":null,"evidence_quote":"Offers the Cartesian evaluative mindset as an alternative to both trust and distrust, which the paper uses to sharpen what healthy distrust adds."},{"cited_title":"On certainty, volume 174","cited_arxiv_id":null,"evidence_quote":"Supplies the point that any distrust must in turn trust something else, used to define what makes distrust 'healthy'."},{"cited_title":"Algorithms","cited_arxiv_id":null,"evidence_quote":"Provides the power-sensitive analysis of how technologies enable and disable practices, which the paper uses to situate healthy distrust in contexts of power."},{"cited_title":"Peters, Ingrid Scharlau, and Barbara Hammer","cited_arxiv_id":null,"evidence_quote":"Empirical survey supporting the claim that an AI's trustworthiness does not imply that users do or should trust the system."}],"review_version":1}