{"id":"2cf8cace-35bb-42c1-b192-014e806cc4ac","arxiv_id":"2604.25456","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs exhibit dialect-dependent biases when recommending restaurants and products, with Mistral-small-3.1 and Llama-3.1 models showing heightened sensitivity to Indian English and code-switched prompts in specific categories.","lead":"The paper tests how prompts in Southern American English, Indian English, and Hindi-English code-switching affect the restaurant and product recommendations produced by several LLMs using balanced lists from public datasets. A smart generalist should read it to see whether everyday language differences can steer AI suggestions in ways that may disadvantage certain user groups.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Prompt length, phrasing, and list presentation may confound attribution of effects to dialect alone","rationale":"This directly extends the reader's weakest assumption with a concrete mechanism (length/structure confounds) that the regression setup does not address. The statistical pipeline is otherwise appropriate, but the isolation of dialect as the sole manipulated variable is the least secure link for the headline claim.","tokens_in":1831,"tokens_out":337,"duration_ms":33162,"concrete_test":"Extract or reconstruct the exact prompt templates for each dialect from the methods section; compute mean token length and number of sentences/clauses across the 20 seeds for each dialect. If lengths differ by >10% or structure varies, re-fit the mixed-effects models with prompt length as an additional fixed covariate and re-run the LRTs for dialect; if the dialect effect loses significance (p>0.05) in the affected models, the attribution weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that observed differences in per-cuisine response counts (tested via mixed-effects regression and LRTs on fixed effects of dialect) are caused by the dialect variation (AE vs. IE vs. CS) rather than incidental prompt differences. The method adds balanced name lists to dialect-varied prompts in a zero-shot setting, but does not report matching or controlling for resulting differences in total prompt length, token count, syntactic structure, or how the list is embedded. If IE or CS prompts systematically alter these surface features, the post-hoc EMM differences and model-specific sensitivities could reflect LLM sensitivity to format or length rather than linguistic bias.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper investigates linguistic biases in LLM-based restaurant and product recommendations by varying prompts across Southern American English (AE), Indian English (IE), and Code-Switched Hindi-English (CS) dialects. Using balanced name lists sampled from the Yelp Open Dataset and Walmart product reviews, the authors zero-shot prompt multiple LLMs (Mistral-small-3.1 and Llama-3.1 family) across 20 seeds to select top-20 items, aggregate per-cuisine and per-category response counts, and fit mixed-effects regression models with likelihood-ratio tests on the dialect fixed effect followed by post-hoc estimated marginal means comparisons.","tokens_in":1985,"tokens_out":716,"duration_ms":43912,"significance":"If the central attribution to dialect holds, the work supplies concrete empirical evidence that prompt dialect can shift LLM recommendation distributions in a zero-shot cold-start setting, with reported sensitivities in Mistral-small-3.1 and Llama-3.1 models to IE/CS prompts for restaurants and in Llama-3.1-70B for certain product categories. The use of real datasets, repeated list sampling over seeds, and standard mixed-effects modeling with LRTs and EMMs constitutes a reproducible statistical pipeline that could be extended to other recommendation domains.","major_comments":[{"comment":"Methods (prompt construction and experimental setup): The procedure adds dialect-varied instructions to balanced name lists but does not report verification that the resulting AE, IE, and CS prompts have equivalent token lengths, total character counts, or syntactic structures. Because the mixed-effects models treat dialect as the sole fixed effect of interest and interpret count differences via LRTs and post-hoc EMMs, any systematic surface-form differences could confound the reported model-specific sensitivities (e.g., Mistral-small-3.1 and Llama-3.1 to IE/CS). This is load-bearing for the claim that dialect itself drives the observed recommendation patterns.","section":"Methods"},{"comment":"Results (statistical reporting): While the mixed-effects regression and LRT approach is defensible for count data, the paper does not supply the full model tables (coefficients, random-effect variances, or exact p-values for all LRTs) or the precise prompt templates used. Without these, it is impossible to confirm that the post-hoc differences attributed to dialect are not artifacts of uncontrolled prompt features or to replicate the exact conditions under which the Llama-3.1-70B shows CS sensitivity in four product categories.","section":"Results"}],"minor_comments":[{"comment":"Abstract: Model names should be given with exact sizes (e.g., Llama-3.1-8B vs. 70B) rather than only 'llama-3.1 family' to allow immediate comparison with the results tables.","section":"Abstract"},{"comment":"Figure and table captions: Ensure all captions explicitly state the number of seeds, the exact dependent variable (aggregate response counts), and the reference level used in the mixed-effects models.","section":"Figures/Tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical contribution to the bias-in-recommendations literature, but the absence of prompt-length controls is a common methodological gap in this subfield; addressing it would strengthen the paper's position relative to prior work on surface-form confounds in LLM prompting."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive and detailed comments on our manuscript. We address each major comment point by point below, agreeing that the suggested additions will improve the clarity and reproducibility of the work.","responses":[{"response":"We agree that explicit verification of prompt surface-form equivalence is important to strengthen the attribution of effects to dialect. Although the restaurant and product name lists are identical across conditions, with variation limited to the dialect-specific instructional prefixes, we did not report token lengths, character counts, or syntactic comparisons in the original submission. In the revised manuscript we will add a dedicated subsection (or appendix) that reports average token counts using each model's tokenizer, total character counts, and a brief qualitative assessment of syntactic structure for the AE, IE, and CS prompt variants. This will demonstrate that any differences are small and not systematically confounded with the dialect factor.","revision_made":"yes","referee_comment":"[Methods] Methods (prompt construction and experimental setup): The procedure adds dialect-varied instructions to balanced name lists but does not report verification that the resulting AE, IE, and CS prompts have equivalent token lengths, total character counts, or syntactic structures. Because the mixed-effects models treat dialect as the sole fixed effect of interest and interpret count differences via LRTs and post-hoc EMMs, any systematic surface-form differences could confound the reported model-specific sensitivities (e.g., Mistral-small-3.1 and Llama-3.1 to IE/CS). This is load-bearing for the claim that dialect itself drives the observed recommendation patterns."},{"response":"We concur that complete model tables and exact prompt templates are required for full reproducibility and to allow independent verification of the statistical claims. The original manuscript described the modeling pipeline at a high level but omitted the detailed outputs and templates. In the revision we will include an appendix containing the full mixed-effects regression tables (fixed-effect coefficients, random-effect variances, likelihood-ratio test statistics, and exact p-values) for all model families and tasks, together with the verbatim prompt templates used for each dialect and recommendation domain. This will enable readers to replicate the precise conditions under which sensitivities (including Llama-3.1-70B CS effects in four product categories) were observed.","revision_made":"yes","referee_comment":"[Results] Results (statistical reporting): While the mixed-effects regression and LRT approach is defensible for count data, the paper does not supply the full model tables (coefficients, random-effect variances, or exact p-values for all LRTs) or the precise prompt templates used. Without these, it is impossible to confirm that the post-hoc differences attributed to dialect are not artifacts of uncontrolled prompt features or to replicate the exact conditions under which the Llama-3.1-70B shows CS sensitivity in four product categories."}],"tokens_in":1649,"tokens_out":602,"duration_ms":67262,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main observation is that Mistral-small-3.1 and the Llama-3.1 models produce different cuisine and category counts when the prompt uses Indian English or Hindi-English code-switching instead of Southern American English. They test this in a cold-start top-20 selection task on balanced name lists drawn from Yelp restaurants and Walmart products, then aggregate counts across 20 seeds and fit mixed-effects models with likelihood-ratio tests and post-hoc comparisons.","headline":"The paper finds some LLMs shift restaurant and product recommendations with Indian English or code-switched prompts, but prompt length and phrasing differences could explain part of the pattern.","tokens_in":2500,"tokens_out":168,"would_cite":false,"duration_ms":43835,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLM restaurant and product recommendations shift depending on the English dialect in the prompt.","keywords":["linguistic bias","LLM recommendations","dialect variation","Indian English","code-switching","restaurant recommendations","product recommendations","AI fairness"],"falsifier":"Re-running the exact same lists and models but with prompts rewritten so that length, structure, and name-list phrasing are identical across all three dialects and then finding no significant dialect effects in the regression would falsify the claim that dialect itself drives the recommendation shifts.","tokens_in":2745,"feed_emoji":"🗣️","tokens_out":573,"duration_ms":47413,"temperature":0.7,"pith_summary":"The paper tests whether large language models produce different recommendations when given prompts written in Southern American English, Indian English, or Hindi-English code-switching. Researchers added balanced lists of restaurant and product names to each prompt and asked several LLMs to pick the top twenty options in a zero-shot setting. They repeated the process across many random list samples and used mixed-effects regression plus likelihood ratio tests to compare how often each cuisine or product category was chosen. The results indicate that dialect influences the distribution of selections, with some models showing stronger shifts under Indian English and code-switched prompts. Because LLMs now handle many everyday recommendation tasks, such differences could mean users receive systematically different suggestions based on how they naturally write.","feed_headline":"LLM Picks Change With English Dialect in Prompts","feed_subtitle":"Certain models favor different cuisines and categories when prompts use Indian English or code-switched Hindi-English rather than Southern U","key_machinery":"Mixed-effects regression on per-category response counts, with dialect type as a fixed effect and random effects for seeds and prompt instances, followed by likelihood ratio tests and pairwise comparisons of marginal means.","core_discovery":"Using fixed balanced lists of names drawn from the Yelp and Walmart datasets, the authors zero-shot prompt multiple LLMs to select top-20 restaurants or products from dialect-varied prompts. Aggregate counts per cuisine and category are then analyzed with mixed-effects regression models whose fixed effects include dialect type; likelihood ratio tests and post-hoc comparisons of estimated marginal means show statistically detectable differences. The Mistral-small-3.1 and both Llama-3.1 models respond to Indian English and code-switched prompts by changing restaurant distributions, while the 70B Llama model is especially sensitive to code-switched prompts in four of seven product categories,偏好","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM Picks Vary With Prompt Dialect","Prompt Dialects Change LLM Restaurant Choices","Indian and Code Switched English Alter LLM Picks","Dialect in Prompts Varies LLM Product Recommendations"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Observed differences in how often each restaurant cuisine or product category is recommended arise only from the dialect features of the prompts and not from any uncontrolled differences in prompt length, sentence structure, or how the models read the inserted name lists.","fun_headline_variants_meta":{"raw":{"variants":["LLM Picks Vary With Prompt Dialect","Prompt Dialects Change LLM Restaurant Choices","Indian and Code Switched English Alter LLM Picks","Dialect in Prompts Varies LLM Product Recommendations"]},"model":"grok-4.3","cost_usd":0.010682,"raw_usage":{"total_tokens":4715,"prompt_tokens":831,"num_sources_used":0,"completion_tokens":47,"cost_in_usd_ticks":106815500,"prompt_tokens_details":{"text_tokens":831,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3837,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":831,"tokens_out":47,"duration_ms":56136,"temperature":1.0,"reasoning_tokens":3837,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-07T16:23:56.599875+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-running the exact same lists and models but with prompts rewritten so that length, structure, and name-list phrasing are identical across all three dialects and then finding no significant dialect effects in the regression would falsify the claim that dialect itself drives the recommendation shifts.","supporting_citations":[],"review_version":1}