{"id":"caff1b6d-7371-4d60-8790-301edc070f4a","arxiv_id":"2501.04763","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Google and Bing organic results and Google's Newsblock tilted toward left-leaning media before the 2024 US election, and Republican-focused queries consistently increased right-leaning sources, while Democratic-query effects were mixed.","lead":"A large-scale audit of Google and Bing search results before the 2024 US election finds that both engines surfaced left-leaning news sources more often than right-leaning ones across all query types. The same data show that searches mentioning Republican candidates returned significantly more right-leaning sources, while the pattern for Democratic queries was weaker and less consistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'left-leaning prioritization' claim depends on counting MBFC 'left-center' outlets (CNN, NYT, et al.) as left-leaning; a stricter coding could erase the absolute effect, though the Republican-query right-leaning overrepresentation may persist.","rationale":"The reader's conditional verdict identifies the MBFC-based measurement as the weakest assumption; I agree and sharpen it: the specific load-bearing step is the binary recoding that folds 'left-center' into 'left' and 'right-center' into 'right.' Because the most prevalent outlets in the data are left-center by MBFC, the absolute left-leaning majority is largely a re-description of mainstream outlet prevalence. The relative partisan-gap finding (H1b) is more robust to this coding change, which is why the paper's most consistent effect may survive a stricter threshold. Still, the headline claim as written—'both search engines tend to prioritize left-leaning media sources'—is not established unless the recoding is shown to be immaterial. A sensitivity analysis with a stricter coding or an independent bias rating would settle the question. Since the authors already present the work as an audit with acknowledged limitations, the appropriate outcome is to keep the conditional verdict: accept only after the measurement robustness check is provided and, ideally, the data/code are released.","tokens_in":23352,"tokens_out":5940,"duration_ms":54700,"concrete_test":"Recompute the descriptive shares and re-estimate the GLMMs in Tables C1–C4 after recoding MBFC labels into three categories—left (left/extreme left), center (least biased/left-center/right-center), right (right/extreme right)—so that only 'left' and 'extreme left' count as left-leaning. If the aggregate left-leaning share no longer exceeds the right-leaning share, or the Republican-query ORs for right-leaning sources materially attenuate, the central claim is an artifact of the binary recoding. A complementary check is to replicate the main models using AllSides ratings for the same domains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the Measures section, the authors recode MBFC's ideological categories into two dummies, with any 'left-center' or 'left' rating counted as left-leaning and any 'right-center' or 'right' rating counted as right-leaning. The top journalistic sources in Tables B2.1/B2.2—CNN, NYT, Washington Post, NBC, ABC, USA Today—are classified by MBFC as left-center, so the aggregate finding that 'roughly half of media sources have at least some leaning toward the political left' is largely a statement about mainstream outlets, not about ideologically left media. If 'left-center' is excluded from the left dummy, the left-leaning share likely drops below the right-leaning share for several collections, directly undercutting the abstract's claim that both engines 'prioritize left-leaning media sources.' The H1b relative effect (right-leaning overrepresentation for Republican queries) is less vulnerable because it compares the same coding across query types, but it too rests on the same unvalidated labels. The paper provides no sensitivity analysis with alternative codings, no inter-rater reliability for the manual source-type labeling, and no released data or code to audit the recoding. This measurement decision is the single most load-bearing premise for the paper's headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper audits Google and Bing before the 2024 US presidential elections, using virtual agents to collect 305,901 search results from June to November 2024 across several US locations. The authors classify sources by type, then for journalistic sources measure political leaning by matching domains to the MediaBiasFactCheck (MBFC) database, recoding MBFC categories into binary left- and right-leaning indicators. They test whether Democratic-focused queries increase the presence of left-leaning media (H1a) and Republican-focused queries increase the presence of right-leaning media (H1b), and they examine moderation by location and time. The findings show that left-leaning sources are generally more prevalent overall, while H1b is consistently supported across engine, collection period, and interface element; H1a is only partially supported, with null results for several daily and Newsblock models. The paper concludes that search engines mirror partisan divides and may contribute to perceived polarization.","tokens_in":23577,"tokens_out":4070,"duration_ms":40125,"significance":"If the findings hold, this is a valuable large-scale algorithmic audit with several genuine strengths: a large and clearly described data collection, controlled virtual agents with randomized simultaneous queries, repeated daily and monthly collections, separate treatment of Google's Newsblock, and mixed-effects models with random intercepts by agent. The relative partisan-gap result for right-leaning media under Republican-focused queries is consistent and robust within the paper's coding, and it is a meaningful contribution to the search-engine-bias literature. The absolute claim that both engines 'prioritize left-leaning media sources' is more exposed to measurement assumptions, but the underlying data collection appears sound and the paper is transparent about many design limitations. These features make the manuscript a plausible candidate for publication after the robustness concerns below are addressed.","major_comments":[{"comment":"The central absolute claim that both search engines 'prioritize left-leaning media sources' is directly determined by the decision to collapse MBFC's 'left-center' and 'left' ratings into a single left-leaning dummy. Because the most frequent sources in Tables B2.1 and B2.2 (cnn.com, nytimes.com, washingtonpost.com, nbcnews.com, etc.) are classified by MBFC as left-center, the descriptive statement that roughly half of media sources have at least some leaning toward the political left is largely a statement about mainstream outlets. The paper provides no sensitivity analysis under alternative codings, no report of the MBFC category distribution in the matched sample, and no release of data or code to audit the recoding. I request a robustness section that (i) reports the share of each MBFC category among matched journalistic domains, (ii) re-estimates the main models excluding left-center and right-center from the leaning dummies or using a three-level coding, and (iii) explicitly states whether the abstract's absolute 'prioritize left-leaning media' claim survives the stricter coding.","section":"Measures"},{"comment":"H1 is only partially supported. The monthly Bing (OR = 1.23, p < 0.001) and Google organic (OR = 1.36, p = 0.005) results support H1a, but the monthly Google Newsblock result is null (OR = 1.05, p = 0.420), and the daily Google organic (OR = 0.96, p = 0.264) and daily Google Newsblock (OR = 1.04, p = 0.063) results do not reach statistical significance. The consistent left-side evidence in the daily data is that Republican queries decrease the presence of left-leaning sources, not that Democratic queries increase it. The abstract's statement that both search engines tend to prioritize left-leaning media sources overstates the evidence and should be rephrased to separate the absolute prevalence of left-leaning sources from the relative partisan-query gap and to acknowledge the pronounced engine-, time-, and interface-dependence of the H1a results.","section":"Results — User-side factors of political information curation"},{"comment":"Source-type classification relies on 'existing lists of domains ... from earlier auditing studies produced by the authors' plus manual labeling by one author, with no reported inter-rater reliability and no publication of the lists. Because the journalistic-source subset is the population on which all MBFC matching and the main models are based, a systematic error in this step could affect which domains enter the analysis and, in principle, the estimated partisan gap. Please provide the classification lists or a transparency appendix with coding statistics and a reliability check (for example, a second coder on a sample of domains).","section":"Measures"}],"minor_comments":[{"comment":"The list of Republican-focused queries appears truncated in the text: after 'electionsdonaldtrump' the string 'jdvance' is not wrapped in quotation marks and the query is incomplete; please correct this typo so that the query set is unambiguous.","section":"Data collection"},{"comment":"The spelling of Diakopoulos is inconsistent: 'Diakopoulous' appears in the introduction and elsewhere, while the reference list uses 'Diakopoulos'; please unify the spelling throughout.","section":"References"},{"comment":"The random-effects grouping variable is labeled 'html_name' in the full regression tables; a more descriptive label such as 'agent_id' would clarify that the random intercepts are at the virtual-agent level.","section":"Appendix Tables C1–C4"},{"comment":"The limitations acknowledged in the Discussion (small query pool, limited geographic locations with almost no battleground states, and analysis of political leaning only for journalistic media) are relevant and should be echoed at the end of the Results section so that readers weigh the descriptive claims accordingly.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The relative partisan-gap finding (H1b: Republican queries consistently increase right-leaning sources) is solid and publishable on its own. The absolute left-leaning-prioritization claim is the headline result, and it is the most exposed to the MBFC recoding decision. A sensitivity analysis with alternative codings would likely settle whether the headline needs to be softened or can stand; without it, the abstract overclaims relative to the paper's own partial support for H1a. I see no circularity or fabrication concerns, and the data collection effort is substantial. The manuscript fits the journal's scope and deserves a chance for revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious, large-scale audit. The team collected 305,901 search results from Google and Bing over the six months before the 2024 US election, covering organic results plus Google Newsblock, with multiple locations and daily data in the final stretch. The most robust finding is H1b: Republican-focused queries consistently increase the share of right-leaning news sources. The odds ratios are sizable and stable — roughly 1.5 for organic results on both engines, and around 2.8–3.1 for Newsblock — and they hold in both monthly and daily collections. That is a real, new empirical result.\n\nThe paper also does several things well. The data collection is carefully described: virtual agents, controlled IP locations, cookie clearing, multiple agents per query to handle randomization. The regression approach (mixed-effects logit with random intercepts for agents) is appropriate. Extending the audit to Google's Newsblock is a genuine addition, since most prior work stops at organic results. The time-series analysis around the election is a nice contribution.\n\nThe soft spot is the headline claim about absolute left-leaning overrepresentation. The authors recode MediaBiasFactCheck categories into binary dummies, counting 'left-center' as left-leaning. Since CNN, NYT, Washington Post, NBC, and ABC are all MBFC 'left-center', the statement that both engines prioritize left-leaning media is largely a statement about mainstream outlets. The stress-test concern is accurate: exclude 'left-center' and the absolute claim likely weakens or flips. The paper offers no sensitivity analysis with alternative codings, and no data or code are released to check the recoding. This does not sink H1b, because that effect compares the same coding across query types, but it does mean the abstract overstates what the data support.\n\nOther issues are more minor. H1a is only partially supported: Democratic queries do not significantly increase left-leaning sources in Google organic daily or Newsblock. Location coverage excludes battleground states, which the authors acknowledge. The manual source-type labeling has no reported inter-rater reliability.\n\nWho should read this: people working on search engine neutrality, political communication, and platform governance. It deserves a serious referee — the scale and the H1b finding justify the time — but the review should push for a sensitivity analysis around the MBFC coding and a more measured abstract. I would engage with it, but I would not cite the absolute left-leaning claim without qualification.","headline":"Large, careful audit whose absolute 'left-leaning prioritization' claim rests on a debatable MBFC recoding, but the query-slant effect (Republican queries increase right-leaning sources) is real and consistent across engines.","tokens_in":24119,"tokens_out":1657,"would_cite":true,"duration_ms":17483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Google and Bing skewed toward left-leaning news sources before the 2024 US election, an audit finds.","keywords":["search engine audit","political information curation","partisan gap","ideological slant","2024 US elections","Google","Bing","media polarization"],"falsifier":"Re-run the audit using an independent measure of source ideology, such as expert panels or a different bias database, and check whether the overrepresentation of right-leaning media for Republican queries—odds ratios around 1.55 in daily organic results—persists. The claim would also be weakened if the left-right imbalance vanished once source quality, recency, or topical relevance were controlled for.","tokens_in":23151,"feed_emoji":"🔍","tokens_out":3997,"duration_ms":33779,"temperature":0.7,"pith_summary":"The paper tries to establish that Google and Bing, audited in the six months before the 2024 US presidential election, systematically prioritized left-leaning journalistic media in top search results, and that the partisan slant of the query shifted the ideological composition of those results. Republican-focused queries consistently raised the share of right-leaning outlets, while Democratic-focused queries raised left-leaning outlets in most but not all conditions. If true, this means search engines are not neutral information gates but active mirrors of the polarized US media environment, with the potential to amplify perceived polarization among users.","feed_headline":"Search engines skewed left before 2024 election","feed_subtitle":"Partisan queries pushed results further apart; right-leaning outlets surged for Republican searches.","key_machinery":"The study uses a virtual agent-based algorithm audit: cloud-hosted Firefox agents searched from fixed US IP addresses, cleared cookies between queries, and repeated the same queries monthly and daily in the run-up to the election. Media sources in the results were classified by type, then matched to MediaBiasFactCheck ratings which were recoded into binary left- and right-leaning dummy variables; generalized linear mixed-effects models with a logit link predicted the likelihood of encountering a left- or right-leaning source as a function of query type, controlling for date and location.","core_discovery":"The central claim is that information curation on Google and Bing during the 2024 US election cycle exhibited a partisan gap: roughly half of all journalistic media results were left-leaning across all query types, and right-leaning media were consistently overrepresented when queries named Republican candidates. Using odds ratios from mixed-effects models, right-leaning sources were 1.49 to 1.66 times more likely to appear in organic results for Republican queries in the monthly collection, and 1.55 to 1.57 times more likely in the daily collection, with even larger effects in Google's Newsblock (odds ratios of 2.79 to 3.13). Democratic queries increased left-leaning sources for Bing and Google organic results in the monthly data but not in the daily data, and never decreased right-leaning sources. The paper interprets this as search engines mirroring rather than correcting the partisan divides present in the US media environment.","pith_inferences":["If the MediaBiasFactCheck-based coding holds, the evidence suggests search engines are not diversifying political information but reproducing the existing imbalance of the US media ecosystem, which could strengthen perceptions of media bias among right-leaning users.","The asymmetric pattern—Republican queries polarize output more than Democratic queries—hints that the engines' response to query slant may be driven by the availability or authority of right-leaning sources, a hypothesis that could be tested in other elections and countries.","A natural extension is to analyze linked article content and tone rather than source-level ideology, to distinguish between visibility and sentiment in search results.","The audit used only cloud IP locations, missing most battleground states; expanding to purple-state locations is a concrete next test for the location-based claims."],"forward_implications":["Search engine users received an ideologically skewed media diet during the 2024 election, with left-leaning outlets dominating and right-leaning outlets surfacing mainly in response to Republican-focused queries.","The partisan gap in organic results was stable across US locations and over time, suggesting it is a structural feature of the engines' curation rather than a localized or transient artifact.","Google's Newsblock behaved differently and more erratically than organic results, flipping its partisan lean on adjacent days, so additional interface elements can amplify or counter the slant of organic results.","Source-level leaning does not directly measure whether a candidate is portrayed positively or negatively, so content-level analysis would be needed to assess tone.","The results give empirical support to public claims of search-engine skew, but they do not establish deliberate manipulation or a specific effect on voters.","Overrepresentation of right-leaning media for Republican queries was consistent across both engines, while Democratic queries never reduced right-leaning sources, suggesting an asymmetric response to query slant."],"supporting_citations":[{"why":"Earlier observation of left-leaning slant in election search results that this study extends to 2024.","marker":"Diakopoulos et al. (2018)"},{"why":"Baseline for US election search behavior and partisan source homogeneity in the 2018 midterms.","marker":"Trielli & Diakopoulos (2022)"},{"why":"Comparative audit of six search engines in the 2020 primaries showing journalistic media dominance.","marker":"Urman et al. (2022a)"},{"why":"German election audit showing that query orientation affects the selection of sources, a key precedent for the query-slant analysis.","marker":"Unkel & Haim (2021)"},{"why":"Evidence that users engage with partisan news on Google Search, providing context for why ideological skew in results matters.","marker":"Robertson et al. (2023)"},{"why":"Longitudinal study of time-based fluctuations in US election search results, supporting the daily and monthly temporal analysis.","marker":"Ulloa et al. (2024a)"},{"why":"Demonstrates location-based personalization of political queries, motivating the search-localization research question.","marker":"Kliman-Silver et al. (2015)"},{"why":"Uses MediaBiasFactCheck ratings, the same labeling source used here for political leaning, lending methodological precedent.","marker":"Baly et al. (2018)"}],"fun_headline_variants":["Search engines skewed left overall, but right-leaning for GOP queries","Audit: Google and Bing showed left-leaning bias in election results","Partisan gap: left-leaning bias overall, right-leaning surge for Republican queries","Search results mirrored US partisan divides before 2024 election","Right-leaning media overrepresented for Republican searches on Google and Bing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes MediaBiasFactCheck's ideological labels, and the binary recoding of those labels into merely left-leaning or right-leaning, accurately measure the political leaning of news sources; if that coding is systematically off, the measured partisan gap could be an artifact of the labeler rather than of the search engines.","fun_headline_variants_meta":{"raw":{"variants":["Search engines skewed left overall, but right-leaning for GOP queries","Audit: Google and Bing showed left-leaning bias in election results","Partisan gap: left-leaning bias overall, right-leaning surge for Republican queries","Search results mirrored US partisan divides before 2024 election","Right-leaning media overrepresented for Republican searches on Google and Bing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3211,"prompt_tokens":979,"completion_tokens":2232,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2138}},"tokens_in":595,"tokens_out":2232,"duration_ms":16833,"temperature":1.0,"reasoning_tokens":2138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:26:46.408314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the audit using an independent measure of source ideology, such as expert panels or a different bias database, and check whether the overrepresentation of right-leaning media for Republican queries—odds ratios around 1.55 in daily organic results—persists. The claim would also be weakened if the left-right imbalance vanished once source quality, recency, or topical relevance were controlled for.","supporting_citations":[],"review_version":1}