{"id":"50b95c71-ed02-4497-992d-f46b4d69d5d6","arxiv_id":"2501.00828","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"On a Queneau-Fénéon corpus, topic variation increases embedding dispersion more than style variation, but the style effect is confounded with human versus GPT-4o authorship.","lead":"This study asks whether a text's writing style changes how language models spread out their meaning vectors, using a French and English literary corpus. It reports that topic and style both affect this dispersion, with topic mattering more, but the design mixes human-written and AI-generated texts in a way that undermines the style result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Style hypotheses (S') and (S'') compare human-authored reference texts to GPT-4o-generated texts, so the reported style-driven dispersion could be a human-vs-machine artifact rather than an effect of style variation.","rationale":"The reader correctly identifies the human-vs-GPT-4o confound as the load-bearing weakness. Every local dispersion hypothesis in Section 4.2, except the global (T-S), compares a human-authored class with a GPT-4o-generated class while claiming to vary only one stylistic or topical dimension. Since GPT-4o generation is known to introduce systematic distributional differences (uniformity, lexical choices, sentence length, and other artifacts), the observed inequalities dbar(QUENEAU_REF) > dbar(QUENEAU_GEN) and dbar(FENEON_GEN) > dbar(FENEON_REF) can be produced without any true style effect. The paper's own Section 4.3 interprets correlations with feature frequencies as evidence for style, but those features also differ between human and machine text, so they do not resolve the confound. The global (T-S) result FENEON_REF > QUENEAU_REF is cleanly between human texts and does support the weaker claim that topic variation has a larger effect than style variation on these two human corpora, but it does not show that style alone matters. The proposed check—comparing two GPT-4o-generated classes that differ only in style variability—would isolate the style effect. If that check fails, the central claim should be weakened to 'topic variation affects embedding dispersion; style effects are confounded and unverified.' The dataset, code, and multi-model methodology are useful contributions, but they do not rescue the central claim as stated. Hence the reader's REJECT verdict should remain unchanged.","tokens_in":14274,"tokens_out":6790,"duration_ms":59990,"concrete_test":"Regenerate from QUENEAU_REF a machine-only style-variation class Q_GEN_VARIED by prompting GPT-4o to rewrite each of the 73 texts in a different Queneau style (using the same per-style target descriptions as the original Exercices), while keeping the topic fixed. Then rerun the Section 4.2 UMAP protocol on Q_GEN_VARIED versus the existing QUENEAU_GEN (both GPT-4o-authored, same topic, uniform style). If the mean centroid distance dbar(Q_GEN_VARIED) is not significantly greater than dbar(QUENEAU_GEN) across the same 12 models and languages, then (S') is an artifact of human-vs-GPT-4o differences. For completeness, similarly generate FENEON_GEN_UNIFORM (GPT-4o rewriting Feneon texts in one uniform style) and check dbar(FENEON_GEN) > dbar(FENEON_GEN_UNIFORM). This removes the authorship confound while keeping the generation pipeline and prompt template otherwise identical.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that style influences embedding dispersion is not identifiable from the reported comparisons. In Section 4.2, hypothesis (S') is tested as dbar(QUENEAU_REF) > dbar(QUENEAU_GEN): QUENEAU_REF is 73 human-written Queneau variations on one topic, while QUENEAU_GEN is GPT-4o's rewriting of those stories in one uniform Feneon style (Section 3.2). The intended manipulated variable is 'number of styles', but authorship (human vs GPT-4o), text length, and generation artifacts change simultaneously; in English, the prompt even imposes 'strictly less than 30 words and using only 1 to 3 sentences', adding a length confound. Likewise (S'') compares GPT-4o-generated FENEON_GEN (varied styles) against human FENEON_REF (uniform style). If GPT-4o outputs have systematically smaller or larger embedding dispersion than human texts—for reasons unrelated to the target style dimension—both (S') and (S'') can hold without any style effect. The only non-confounded comparison, the global (T-S) test dbar(FENEON_REF) > dbar(QUENEAU_REF), is between two human literary corpora and can support topic dominance, but it does not establish that style alone changes dispersion because it never varies style while holding authorship, topic, and generation procedure fixed. Consequently, the paper's evidence for 'writing style influences embedding dispersion' does not withstand the human-vs-machine confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the QUENEAU-FENEON corpus, built from two French literary works and their English translations, with two human reference classes (QUENEAU_REF: 73 same-topic, multi-style texts; FENEON_REF: 73 multi-topic, same-style texts) and two GPT-4o-generated classes (QUENEAU_GEN: same topic, uniform Fénéon style; FENEON_GEN: varied topics, varied Queneau styles). For twelve embedding models and two languages, the authors measure within-class dispersion as the mean Euclidean distance to the class centroid in UMAP-reduced embedding space, and test hypotheses that topic variation and style variation increase dispersion, with topic having the larger effect. They also correlate per-text dispersion differences with differences in eight stylistic feature groups. The paper concludes that writing style influences embedding dispersion, though topic variation has a stronger effect.","tokens_in":14566,"tokens_out":6981,"duration_ms":64595,"significance":"If the central claim were established, the paper would offer a compact, reproducible framework for comparing how different embedding models encode style versus topic: the corpus construction is transparent, the experiments span twelve models and two languages, and code and data are promised in a GitHub repository. The one clean human-human comparison, (T-S) with dbar(FENEON_REF) > dbar(QUENEAU_REF), is a reasonably supported demonstration that topic variation is associated with larger dispersion than style variation in these two literary corpora. However, the paper's stronger claim that style alone drives dispersion is not identifiable from the reported comparisons, because the style hypotheses are tested on human-versus-machine comparisons that also change authorship, generation artifacts, and, for English, text length.","major_comments":[{"comment":"The two style hypotheses are tested on comparisons that vary authorship and generation procedure alongside style. (S') compares QUENEAU_REF (73 human-written Queneau exercises) with QUENEAU_GEN (GPT-4o rewrites of the same stories in a single Fénéon style); the English prompt additionally imposes a strict length constraint ('strictly less than 30 words and using only 1 to 3 sentences', Figure 1). (S'') compares FENEON_GEN (GPT-4o rewrites in varied styles) with human FENEON_REF. If GPT-4o outputs have systematically different embedding dispersion from human texts for reasons unrelated to style, both inequalities can hold without any style effect, so the Section 5 conclusion that writing style influences embedding dispersion is not identifiable from these comparisons. A control that holds authorship and generation fixed, for example GPT-4o rewrites of the FENEON_REF texts in one uniform style versus GPT-4o rewrites in many styles, is needed.","section":"§4.2, hypotheses (S') and (S'')"},{"comment":"The topic hypotheses are also tested on human-versus-machine comparisons: (T') compares GPT-4o-generated FENEON_GEN to human QUENEAU_REF, and (T'') compares human FENEON_REF to GPT-4o-generated QUENEAU_GEN. The same authorship confound therefore applies, and the English QUENEAU_GEN texts are short by instruction. The only fully human comparison, (T-S), supports the ordering FENEON_REF > QUENEAU_REF, but it does not by itself disentangle topic from author and text-length differences. The paper would be substantially strengthened by a generated-control pair in which only topic variability changes while style and generation procedure are held fixed.","section":"§4.2, hypotheses (T') and (T'')"},{"comment":"The interpretability analysis computes Pearson correlations between Δd(X,Y) = d_X(i) − d_Y(j) and Δf_s(X,Y) = f_X^s(i) − f_Y^s(j) without specifying a correspondence between the i-th vector of class X and the j-th vector of class Y. Since the two classes have no natural pairing, it is unclear how a single correlation is obtained; if all pairwise differences are pooled, the effective sample is larger than the 73 texts per class and the reported significance levels are not valid pairwise tests. This makes the feature-dispersion correlations in Figure 4 difficult to interpret and weakens the interpretability evidence for the style effect.","section":"§4.3, Eqs. (8)-(9)"}],"minor_comments":[{"comment":"The French prompt for QUENEAU_GEN contains the typo 'Ré écris'; it should be 'Réécris'.","section":"Figure 1"},{"comment":"The sentence 'we created a generated corpus by tranforming these original texts' contains a typo: 'tranforming' should be 'transforming'.","section":"Section 3"},{"comment":"The reported ranking order '2D PCA, 3D PCA, 10D PCA, 5D PCA' is inconsistent with the mean scores in Table 2: for French, 10D PCA (0.6748) is higher than 3D PCA (0.6117) and even 2D PCA (0.6623); the text should describe the order actually implied by the combined means.","section":"Section 4.1, Table 2"},{"comment":"The model name is written 'xml-roberta-large' once in the paragraph after Table 2, while the table, the list of models, and the references use 'xlm-roberta-large'.","section":"Table 2 and surrounding text"},{"comment":"Significance levels are reported without any multiple-comparison correction across twelve models, two languages, and four local hypotheses; a note on the false-discovery rate or a justification for not correcting would improve the reliability of the pattern reported.","section":"Section 4.2, Table 3"},{"comment":"The Limitations section does not mention the human-versus-machine confound that affects the style and topic hypotheses, nor the length constraint in the English QUENEAU_GEN prompt; both are central to interpreting the results and should be acknowledged.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is technically competent and well-documented, and the global human-human comparison is a reasonable finding. However, the paper's headline claim about style influencing dispersion is currently not identifiable because the style hypotheses rely on human-versus-GPT-4o comparisons with concurrent length and generation differences. This is a fixable problem within the paper's experimental scope: adding machine-generated control classes that vary only the target dimension, or human-human controls that hold authorship fixed, would address the objection. I therefore recommend major revision rather than rejection, contingent on new control experiments and a revised interpretation of the existing comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a look for its corpus, though its central claim does not survive the design. The QUENEAU-FENEON dataset is a thoughtful instrument: Queneau's many-styles/one-topic texts and Feneon's many-topics/one-style texts give a natural way to separate the two dimensions, and adding GPT-4o rewrites extends it into a 2x2. Testing 12 embedding models in French and English, with code and data released, is also real work. The global comparison between the two human reference classes (FENEON_REF vs QUENEAU_REF) is clean and supports topic dominance: topic variation spreads embeddings more than style variation. That result is solid as far as it goes. The soft spots are load-bearing. Every local hypothesis, both style and topic, compares a human-authored class to a GPT-4o-generated class. For style, S' compares human Queneau (many styles) to machine Queneau (one style); S'' compares machine Feneon (many styles) to human Feneon (one style). In both cases, the manipulated variable is style, but authorship, text length, and generation artifacts change at the same time. The English prompt for QUENEAU_GEN even imposes a strict word and sentence limit, adding a length confound. So the observed dispersion differences could be a human-vs-machine artifact rather than a style effect. The topic hypotheses (T) have the same problem. This means the paper's stated conclusion that 'writing style influences embedding dispersion' is unproven. The statistical analysis also leans on post-hoc selection: 2D UMAP was chosen after it gave the best validation scores, and the t-tests treat distances that are not independent (each generated text is a rewrite of a specific original, and the 30 UMAP runs share the same underlying embeddings). These are not minor wrinkles; they cut the foundation out of the local claims. Still, the global topic-superiority result and the corpus itself are genuinely useful resources. The interpretability section reports mostly weak correlations but is honestly labeled. The limitation section is candid about sample size and omitted higher-dimensional UMAP results. Who is this for? Researchers in stylometry and embedding interpretability may want the corpus and the clear negative lesson: generation-based style manipulations need authorship controls. As submitted, it should not be accepted as evidence for the style claim. But it deserves a serious referee, because the underlying question is important and the instrument is close to being able to answer it with a redesigned control. I would engage with it and push for a revision that fixes the confound.","headline":"A carefully built corpus for separating style and topic, but the style-specific conclusion is unidentifiable because every style comparison confounds authorship with GPT-4o generation.","tokens_in":15113,"tokens_out":2849,"would_cite":false,"duration_ms":27770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Writing style measurably changes the spread of text embeddings in language models, but topic variation spreads them more.","keywords":["embedding dispersion","writing style","topic modeling","language models","UMAP","stylometry","text embeddings","multilingual embeddings"],"falsifier":"One concrete test is to generate rewrites that preserve both topic and style while changing only surface wording, then measure whether mean centroid distance changes as much as it does in the style- or topic-change conditions; if it does, dispersion is responding to text generation itself rather than to style or topic. A second test is to run the same dispersion comparisons on a human-authored corpus with topic and style varied independently, without any machine rewrites.","tokens_in":14066,"feed_emoji":"🖋️","tokens_out":5651,"duration_ms":52513,"temperature":0.7,"pith_summary":"The paper asks whether writing style leaves a measurable trace in the geometry of embedding vectors, and whether topic does so more strongly. It builds a bilingual corpus of literary texts in which topic and style are swapped independently: one source has one topic repeated in many styles, another has many topics in one style, and language-model rewrites create the crossed combinations. Measuring mean distance of each text from its class centroid in UMAP projections, it finds that both topic variation and style variation increase embedding dispersion, with topic variation producing larger increases. The authors interpret this as evidence that style is encoded in embedding spaces, but secondary to topic.","feed_headline":"Style shifts embedding geometry—topic shifts it more","feed_subtitle":"Across twelve language models in two languages, topic variation spreads embeddings further than style variation.","key_machinery":"The central object is the QUENEAU-FENEON corpus, a four-cell design in which topic and style vary independently. Each language has 73 texts in each of: same topic with varied styles, varied topics with the same style, and two language-model-generated crossed classes. The central metric is the mean Euclidean distance from each text embedding to its class centroid in UMAP-reduced space, averaged over 30 random seeds. The hypotheses are inequalities between these means: topic variation should increase dispersion, style variation should increase dispersion, and the topic gap should exceed the style gap. This metric converts the abstract question of whether embeddings encode style into a directly testable ordering of numbers.","core_discovery":"The central claim is stated in the conclusion: writing style influences embedding dispersion, and topic variation has a stronger effect. Concretely, the paper predicts and observes the order $$\\bar d_{\\mathrm{FENEON\\_GEN}} > \\bar d_{\\mathrm{FENEON\\_REF}} > \\bar d_{\\mathrm{QUENEAU\\_REF}} > \\bar d_{\\mathrm{QUENEAU\\_GEN}}$$ in mean centroid distance, with pairwise differences significant at the .01 level across most models and both languages. The local hypotheses (T) and (S) attribute the first two gaps to topic heterogeneity and the style gap to style heterogeneity, while the global hypothesis (T-S) attributes the FENEON_REF-to-QUENEAU_REF gap to topic dominating style. Attempted interpretability links dispersion to readability and complexity indexes, function words, and punctuation.","pith_inferences":["If this dispersion signature is stable, embedding dispersion could serve as an unsupervised proxy for detecting whether a text has been stylistically rewritten, without labelled training data.","The same four-cell design could be applied to typologically distant languages; the relative strength of the topic effect may shift if style is carried by different surface features.","A direct extension would replace the mean-centroid-distance scalar with shape descriptors, such as variance along principal axes, to separate style- and topic-specific directions rather than collapsing them into one number.","The English rewriting prompt imposed a word limit not present in the French prompt, so an immediate replication with identical prompt constraints across languages would test whether the observed French-English gap is linguistic or procedural."],"forward_implications":["Stylistic differences among authors are detectable in embedding geometry, so style-sensitive tasks such as authorship verification should treat embedding similarity as carrying style information.","Topic must be controlled before interpreting dispersion as a style signal in any embedding-based analysis.","Models vary in how strongly they encode style; applications that rely on style, such as style transfer evaluation, should not assume all embedding models respond equally.","Readability and complexity indexes, function words, and punctuation are the surface features most associated with the style-driven dispersion.","Translation can dampen the stylistic signal, so multilingual style comparisons need to check that feature frequencies survive translation."],"supporting_citations":[{"why":"Supplies the reference class with one fixed topic and many styles.","marker":"(Queneau, 1947)"},{"why":"Supplies the reference class with many topics and one fixed style.","marker":"(Fénéon and Halperin, 1970)"},{"why":"Supplies the generation model used to create the crossed corpus classes.","marker":"(Achiam et al., 2023)"},{"why":"Supplies the UMAP dimensionality reduction used for dispersion measurement.","marker":"(McInnes et al., 2018)"},{"why":"Supplies the stylistic feature extraction framework used for interpretability.","marker":"(Terreau et al., 2021)"},{"why":"Supplies the benchmark used to select the twelve embedding models.","marker":"(Muennighoff et al., 2022)"}],"fun_headline_variants":["Style shifts embeddings, but topic shifts them more","Topic variation stretches embedding spaces further than style","Twelve models: topic effect on embedding spread > style effect","Embedding dispersion order: topic heterogeneity dominates style","Style matters, but topic matters more for embedding geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire attribution rests on the assumption that the rewriting step changes only the intended dimension, style or topic, and does not systematically alter embedding geometry for other reasons.","fun_headline_variants_meta":{"raw":{"variants":["Style shifts embeddings, but topic shifts them more","Topic variation stretches embedding spaces further than style","Twelve models: topic effect on embedding spread > style effect","Embedding dispersion order: topic heterogeneity dominates style","Style matters, but topic matters more for embedding geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1530,"prompt_tokens":783,"completion_tokens":747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":673}},"tokens_in":399,"tokens_out":747,"duration_ms":7368,"temperature":1.0,"reasoning_tokens":673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:41:43.279952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test is to generate rewrites that preserve both topic and style while changing only surface wording, then measure whether mean centroid distance changes as much as it does in the style- or topic-change conditions; if it does, dispersion is responding to text generation itself rather than to style or topic. A second test is to run the same dispersion comparisons on a human-authored corpus with topic and style varied independently, without any machine rewrites.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reference class with one fixed topic and many styles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reference class with many topics and one fixed style."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the stylistic feature extraction framework used for interpretability."}],"review_version":1}