{"id":"47ad9c35-f1f1-4fef-9b86-cdcd06b5f308","arxiv_id":"2505.09478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated aggregate card sortings agree moderately with human results (NMI around 0.68), with accuracy decreasing for larger card sets and complex labels.","lead":"This paper tests whether large language models can mimic how people organize information in card sorting studies, using 28 real studies with 1,399 participants. It finds moderate agreement, suggesting AI can provide quick early feedback but should not replace human participants.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P3's headline NMI of 0.68 is never calibrated against chance or against human-to-aggregate agreement, so the central claim of 'reasonable agreement' remains underdetermined.","rationale":"Stress-testing the paper in good faith, the empirical core is solid in design: 28 real third-party studies, 1,399 participants, four LLMs, four prompt variants, repeated trials for GPT-4o, and identical processing where the pipeline is applicable. The relative claims (P3 > P1; model differences small; context not helpful; complexity hurts) are reasonably supported by non-parametric tests. The concern is not internal inconsistency but external interpretability of the headline absolute NMI. A value of 0.68 is only meaningful relative to some baseline. The paper reports neither a chance baseline, despite NMI being non-negative and dependent on K and cluster-size distributions, nor a human-reliability baseline, despite the ground truth being a constructed aggregate clustering. The reader's weakest_assumption about the MDS+k-means ground truth is closely related; I would sharpen it: even if the ground-truth clustering is accepted as standard practice, the absence of a human-to-aggregate consistency measure leaves the 'reasonable degree' claim uncalibrated. The proposed test would settle whether the concern actually affects interpretation: if typical human participants agree with the aggregate no better than the LLM does, the claim is conservative; if they agree much better, the claim overstates LLM performance. Given the paper already frames the simulator as augmentation rather than replacement and explicitly claims only moderate agreement, the appropriate verdict remains conditional; neither acceptance nor rejection is warranted by this review pass.","tokens_in":26323,"tokens_out":7101,"duration_ms":83192,"concrete_test":"Run a single calibration analysis on the same 28 studies: (i) compute the 95th percentile of NMI between the human ground-truth clustering and 1,000 random partitions matched for K and cluster-size distribution; (ii) compute the mean NMI between each real participant's own categorization and the human ground-truth clustering. Then compare P3's mean NMI (0.68) with both reference values. If P3 is close to the chance percentile or substantially below the typical human-to-aggregate NMI, the 'reasonable agreement' claim should be weakened; if P3 is near or above typical human-to-aggregate agreement, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on interpreting NMI ≈ 0.68 as 'reasonable agreement' with real card sorting. Two missing calibrations make this interpretation insecure. First, no chance baseline is reported: NMI is a positive information-theoretic quantity, and with the observed K (median 8 clusters) and relatively small card counts (median 34), random partitions with matched marginals can produce non-negligible NMI; the paper provides no permutation test showing that P3's score is far above that floor. Second, no human-consistency benchmark is reported: the ground truth is not raw participant data but a single k-means/MDS clustering with K chosen by a knee rule (§4.5, Eq. 1). We never learn how much an average real participant agrees with that aggregate clustering, so 0.68 has no reference point. If typical participant-to-aggregate NMI were, say, 0.55 or 0.75, the same 0.68 would mean very different things. The relative claims (P1 worse, complexity modulates accuracy, model differences small) are less affected; the absolute claim of a 'good degree of agreement' in the abstract and §5.1 is what lacks calibration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Card Sorting Simulator, an LLM-based prototype that generates synthetic card sorting results, and evaluates it against 28 real-world open card sorting studies (1,399 participants). Four prompt variants are compared: raw-data simulation (P1), similarity-matrix generation (P2), aggregate clustering generation (P3), and clustering without context (P4), across GPT-4o, Gemini, Claude, and DeepSeek. The central empirical claims are that aggregate clustering prompts (P3) achieve a mean NMI of 0.68 against human-derived clusterings, that raw simulation (P1) is markedly worse, that study complexity lowers agreement, and that model choice and prompt context have only minor effects. The authors conclude that LLM-generated card sorting can provide useful preliminary feedback but should not replace human participants.","tokens_in":26538,"tokens_out":6086,"duration_ms":67374,"significance":"The paper's main strength is its ecological validity: the evaluation uses diverse, practitioner-created card sorting studies rather than laboratory tasks, and the multi-prompt, multi-model comparison is well suited to inform tool design. The statistical analyses (nonparametric tests, repeated-measures comparisons) are generally appropriate, and the authors are careful to check output errors and variability. If the absolute agreement claim can be properly calibrated, the paper would be a valuable empirical foundation for AI-augmented information architecture research, and it would extend the sparse prior work on LLM-simulated card sorting. The distinction between abstract clustering generation and low-level participant simulation is a useful design insight, as is the finding that adding readily available context did not improve results.","major_comments":[{"comment":"The headline claim that P2/P3 reach NMI ≈ 0.68 and therefore show 'good degree of agreement' is not calibrated against chance. NMI as computed by scikit-learn is not chance-adjusted, and with the study-level median of about 8 clusters and median 34 cards, two independent random partitions with fixed marginal cluster sizes can attain a non-negligible expected NMI (roughly 0.3 under a chi-square approximation). The paper reports no permutation null distribution or any chance baseline, so the absolute interpretation of 0.68 is underdetermined. Please add a permutation test that shuffles card labels within the ground-truth cluster sizes, report Adjusted Mutual Information, or otherwise quantify how far P3's NMI is above the chance floor. The relative comparisons (P1 vs. P2/P3, complexity effects) are less affected, but the abstract's and §5.1's absolute claim of 'good agreement' depends on this calibration.","section":"§5.1, Table 1"},{"comment":"The ground truth is not raw human sorts but a derived clustering obtained by converting participant sorts into a similarity matrix, applying MDS, and running k-means with K selected by a knee rule. Agreement scores compare LLM clusterings to this constructed aggregate. To interpret NMI = 0.68 as 'reasonable agreement,' the paper needs a human-level reference: the same pipeline should be applied to individual real participants, and the average NMI/ARI between each participant's sort and the aggregate should be reported. This quantity is computable from the already-available dataset. Without it, the same 0.68 could mean that the LLM is near the typical human-to-aggregate consistency or far below it; the current text cannot distinguish these.","section":"§4.5 and Figure 4"},{"comment":"The knee/elbow method automatically selects K for every clustering, including the LLM-generated ones, and the reported NMI, ARI, and edit-distance values depend on these K choices. There is no sensitivity analysis showing that the main conclusions are stable with respect to K determination. Because the complexity results in §5.4 also rely on the same pipeline, a sensitivity analysis over a plausible range of K, or a comparison with an alternative clustering/K-selection method, would materially strengthen the quantitative claims.","section":"§4.5, Eq. (1), §5.4"}],"minor_comments":[{"comment":"The test name is misspelled as 'Willcoxon' and should be 'Wilcoxon.' In addition, the sample sizes n = 54 for P1/P2 and n = 224 for P3 are not explained; please clarify how many studies and repeated trials contribute to each comparison, since Table 1 text implies four GPT-4o trials per study.","section":"§5.1"},{"comment":"The table header 'LLM output format' does not match the rows, which are model names. Rename the header to 'Model'.","section":"Table 3"},{"comment":"The table layout is ambiguous: each row appears to report the averaged within-study standard deviation for four metrics, but the column headers are visually split so that 'mean SD' appears as a single block. Please format the table so that each metric has a distinct mean and SD column, matching the text's description.","section":"Table 2"},{"comment":"The observation that P1 produced completely identical output for all simulated participants in 5 of 28 studies is an important failure mode that should be mentioned in the limitations or discussion, not only in the qualitative results.","section":"§5.1, Figure 5"},{"comment":"The prompts were refined through an iterative process, but the manuscript does not describe which card sorting studies were used in that development process or whether any of the 28 evaluation studies were also used to tune prompts. Please clarify this to rule out potential leakage from prompt engineering into the reported results.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.HC and the empirical dataset is a real strength. The main issue is calibration of the absolute NMI claim, which is addressable with analyses that require no new data collection. I would not reject, but the authors should be asked to add the chance baseline and the human-to-aggregate benchmark before the central 'good agreement' claim is considered established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. This is the first large-scale multi-model, multi-prompt evaluation of LLM-driven card sorting, with 28 real studies and nearly 1,400 participants. The concept was shown earlier by Sauro et al. on a single e-commerce sort; this paper turns that into a proper benchmark and adds several genuinely useful findings: aggregate clustering prompts (P3) beat raw-data simulation (P1), complexity of the card set (card count, label difficulty) reliably degrades agreement, and model choice and contextual prompt details make little difference. The experimental design is careful: non-parametric tests, repeated trials to check variability, quality review of studies, and a sensible pipeline that processes real and synthetic data identically. Credit where due: this is reproducible in structure, even if the underlying study data is proprietary and the code isn't released.\n\nThe main soft spot is calibration of the absolute claim. The paper says NMI ≈ 0.68 indicates 'good agreement' with real participants, but it never reports a chance baseline for NMI or a human-consistency benchmark. With median 34 cards and median 8 clusters, random partitions with matched marginals can produce non-trivial NMI; and the ground truth is a constructed MDS + k-means clustering with K chosen by a knee rule, not raw participant data. We never learn how much an average real participant agrees with that aggregate clustering. If typical participant-to-aggregate NMI is 0.55, then 0.68 is impressive; if it's 0.75, then the LLM is doing worse than the average human. So the relative claims—P1 worse, complexity matters, model differences small—are well supported, but the absolute 'good degree of agreement' in the abstract and §5.1 is underdetermined. That's a fixable problem: a permutation test and an average-participant-vs-aggregate NMI would settle it.\n\nMinor concerns: prompts were iteratively engineered by the authors on card-sorting tasks, which could inflate performance relative to off-the-shelf prompting; the dataset is western/English-only; and LLM outputs needed manual preprocessing, which is acknowledged. These are limitations, not flaws.\n\nWho's this for: UX researchers and tool builders working on AI-augmented research methods. It deserves a serious referee and the authors should be pushed to add the missing calibrations. Worth citing for the empirical benchmark.","headline":"A solid, large-scale empirical benchmark of LLM card sorting, but the headline NMI numbers need calibration against chance and human consistency before we know how good 0.68 really is.","tokens_in":27069,"tokens_out":1798,"would_cite":true,"duration_ms":18239,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated card sorting can reproduce the strongest grouping patterns of real participants, with mean normalized mutual information of 0.68 when the model is asked directly for an aggregate clustering, and performs markedly worse when…","keywords":["card sorting","large language models","synthetic participants","information architecture","user research","mental models","clustering","generative artificial intelligence"],"falsifier":"Re-run the evaluation with an independent reference: compare LLM outputs against the clusters participants themselves named or intended, or replace k-means with hierarchical best-merge clustering on the same similarity matrices. If the P2/P3 advantage over P1 shrinks or the 0.68 mean NMI drops substantially, the reported agreement depends on the chosen analysis pipeline rather than on genuine alignment of mental models.","tokens_in":26121,"feed_emoji":"🗂️","tokens_out":9991,"duration_ms":94609,"temperature":0.7,"pith_summary":"Card sorting asks users to group labeled items so designers can infer the mental model behind an information architecture. This paper argues that a large language model can stand in for that process well enough to give useful early feedback: across 28 real open card sorting studies with 1,399 participants, the best prompt design produced clusterings with a mean normalized mutual information of 0.68 against clusterings derived from human sorts. The strongest result is that the simulation should be prompted to produce one aggregate clustering rather than to imitate individual participants; the aggregate-level prompts (P2 and P3) both reach 0.68, while raw-data simulation (P1) reaches only 0.50. Agreement falls when studies have more cards or harder labels, and neither model choice nor added study context changes the outcome much. The paper positions the simulator as an augmenting tool for quick preliminary feedback, not a replacement for human participants.","feed_headline":"68 percent: LLMs mimic the broad groupings humans make in card sorting","feed_subtitle":"Asking an LLM for one aggregate clustering beats simulating individual participants across 28 real studies.","key_machinery":"The load-bearing component is the Card Sorting Simulator's prompt template, with four output-format variants; the decisive variant is P3, Clustering Generation, which instructs the LLM to output a single best aggregate categorization as if it were the result of analyzed responses from a sample of participants. The comparison pipeline converts all real and simulated sorts into a common representation: a similarity matrix of pairwise co-grouping frequencies, reduced by multidimensional scaling, then clustered with k-means where the number of clusters $K$ is fixed by the knee/elbow point of the within-cluster sum-of-squares curve. Agreement between synthetic and human clusterings is measured with normalized mutual information as the primary metric, alongside adjusted Rand index, edit distance, and Mantel matrix correlation. The paper also includes qualitative inspection of similarity-matrix heatmaps to explain structural divergences such as phantom groupings and over-general category labels.","core_discovery":"The central discovery is that LLM-generated card sorting agrees with human card sorting on the most salient groupings, while disagreeing about individual card placement, category abstraction level, and diversity. Using the study's own evaluation pipeline, Clustering Generation (P3) reaches a mean NMI of 0.68 (SD 0.12) and Similarity Matrix Generation (P2) also reaches 0.68 (SD 0.11), whereas Raw Data Simulation (P1) reaches only 0.50 (SD 0.15); ARI, edit distance, and matrix correlation follow the same ordering, with P3 and P2 close and P1 markedly worse. The authors interpret this as support for their hypothesis H1a, that the more abstract prompt format is the more accurate one, and H1b, that agreement is partial and not sufficient for substitution. Additional findings are that agreement is stable across repeated runs, that differences between Claude, Gemini, GPT-4o, and DeepSeek are small and mostly non-significant, and that adding contextual information (demographics, instructions, welcome messages) does not improve agreement. Study complexity modulates accuracy: mean NMI for P3 falls from 0.77 in low-card-count studies to 0.65 in high-card-count studies, and from 0.78 for simple labels to 0.64 for complex labels.","pith_inferences":["A natural extension the paper does not test is whether the 0.68 agreement is stable under a different ground-truth pipeline: comparing LLM outputs directly with participants' own named categories, or using hierarchical clustering instead of MDS plus k-means, could show whether the reported ordering of P1, P2, and P3 is an artifact of the evaluation pipeline.","The failure of raw-data simulation suggests LLMs store typical associations rather than individual variation; injecting explicit diversity constraints, for example forcing several distinct sorting rationales per simulated participant, might close part of the gap and would be a cheap testable variant.","The complexity effect implies a practical calibration rule: for card sets above roughly 45 cards or with domain-heavy labels, the simulator's output should be labelled as a first-pass hypothesis and weighted less in architecture decisions.","If LLM simulation continues to improve with model generations, the same 28-study benchmark could be rerun as a longitudinal check on whether the measured ceiling is a property of current models rather than of the task itself."],"forward_implications":["If the central claim is right, UX researchers can obtain meaningful preliminary card-sorting feedback before recruiting participants, by prompting an LLM for an aggregate clustering rather than raw individual sorts.","The low-level P1 simulation is reliably worse, so tool builders should avoid spending tokens on generating fake individual participants and instead target aggregate outputs.","Because P3 and P2 perform alike but P2 needs a larger output, clustering generation is the practical default for an automated card-sorting assistant.","Since model choice and prompt context barely move the scores, implementation can prioritise cost, speed, and output reliability over model selection.","Agreement degrades with study complexity, so simulations from large or difficult card sets should be treated as more tentative than those from small, clear sets."],"supporting_citations":[{"why":"Provides the only prior direct comparison of ChatGPT card sorting with a real aggregation of 200 participants, motivating the authors' benchmark and prompt-design approach.","marker":"Sauro et al. (2024)"},{"why":"Supplies the card-sorting analysis methodology of multidimensional scaling with k-means on similarity data that the evaluation pipeline adopts.","marker":"Macías and Culén, 2021"},{"why":"Provides the knee/elbow detection method used to choose the number of clusters in the k-means step.","marker":"Satopaa et al., 2011"},{"why":"Supports the expectation that LLM-generated behavioural data can resemble human responses, framing the first research question.","marker":"Hämäläinen et al., 2023"},{"why":"SimUser is the prior LLM user-simulation tool whose observed limitations inform the persona prompting and the discussion of diversity.","marker":"Xiang et al., 2024"}],"fun_headline_variants":["LLM card sorting: broad agreement, fine detail off","LLMs sort cards like humans, but only at a glance","AI card sorting: matches big picture, misses specifics","Card sorting AI: 0.68 NMI, but details diverge","LLM card sorter: good on trends, weak on particulars"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison depends on treating the statistically derived grouping of the real participants' sorts as ground truth; if that derived grouping is not what participants themselves meant, then every agreement score with the simulated sorts loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["LLM card sorting: broad agreement, fine detail off","LLMs sort cards like humans, but only at a glance","AI card sorting: matches big picture, misses specifics","Card sorting AI: 0.68 NMI, but details diverge","LLM card sorter: good on trends, weak on particulars"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":3054,"prompt_tokens":1088,"completion_tokens":1966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":1879}},"tokens_in":704,"tokens_out":1966,"duration_ms":12527,"temperature":1.0,"reasoning_tokens":1879,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:30:01.683352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with an independent reference: compare LLM outputs against the clusters participants themselves named or intended, or replace k-means with hierarchical best-merge clustering on the same similarity matrices. If the P2/P3 advantage over P1 shrinks or the 0.68 mean NMI drops substantially, the reported agreement depends on the chosen analysis pipeline rather than on genuine alignment of mental models.","supporting_citations":[{"cited_title":"Comparing chatgpt to card sorting results, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the only prior direct comparison of ChatGPT card sorting with a real aggregation of 200 participants, motivating the authors' benchmark and prompt-design approach."}],"review_version":1}