Pith. sign in

REVIEW 2 cited by

Taking Advice from ChatGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.11888 v3 pith:GV72YVPK submitted 2023-05-11 cs.HC

Peter Zhang This is my paper · ORCID
classification cs.HC
keywords advicechatgptparticipantsaccuracyadvisoralgorithmchatbotreceive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A growing literature studies how humans incorporate advice from algorithms. This study examines an algorithm with millions of daily users: ChatGPT. In a preregistered study, 118 student participants answer 2,828 multiple-choice questions across 25 academic subjects. Participants receive advice from a GPT model and can update their initial responses. The advisor's identity ("AI chatbot" versus a human "expert"), presence of a written justification, and advice correctness do not significantly affect weight on advice. Instead, participants weigh advice more heavily if they (1) are unfamiliar with the topic, (2) used ChatGPT in the past, or (3) received more accurate advice previously. The last two effects -- algorithm familiarity and experience -- are stronger with an AI chatbot as the advisor. Participants that receive written justifications are able to discern correct advice and update accordingly. Student participants are miscalibrated in their judgements of ChatGPT advice accuracy; one reason is that they significantly misjudge the accuracy of ChatGPT on 11/25 topics. Participants under-weigh advice by over 50% and can score better by trusting ChatGPT more.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

    cs.AI 2026-07 conditional novelty 6.0 of 10

    HCRA turns collaborative recommendation into a stochastic game whose termination condition is driven by human-calibrated acceptance probability and linguistic self-reflection, yielding higher success rates than uncali...

  2. Do people rely on ChatGPT more than their peers to detect deepfake news?

    econ.GN 2026-08 conditional novelty 5.0 of 10

    In a lab deepfake-detection task, students shifted more toward ChatGPT's advice than toward peers' advice (weight-of-advice 0.59 vs 0.33), though in 2025 sessions they trusted linguistic experts slightly more than ChatGPT.

Pith tools