Modeling Professionalism in Expert Questioning through Linguistic Differentiation

Chung-Chi Chen; Giulia D'Agostino

REVIEW 4 major objections 5 minor 14 references

Modeling Professionalism in Expert Questioning through Linguistic Differentiation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that professionalism in expert questioning is a learnable, interpretable linguistic pattern that can be modeled with a small set of structural and pragmatic features, and that question origin serves as a valid proxy for it.

desk verdict The dataset and the shallow classifier are real contributions, but the paper's central proxy claim is not supported by its own correlation table. read the letter →

arxiv 2507.20249 v1 pith:EO33Z76O submitted 2025-07-27 cs.CL

Giulia D'Agostino , Chung-Chi Chen This is my paper · ORCID

classification cs.CL

keywords professionalismexpertquestioninglinguisticfeaturesearningscallsquestionoriginclassificationinterpretablemachinelearningLLM-generatedtextdetectiondiscoursepragmatics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that professionalism in expert questioning is not an elusive quality but a measurable linguistic pattern. It introduces an annotation scheme for structural and pragmatic features of financial analyst questions—discourse regulators, prefaces, question types, and request types—and shows that these features correlate in the same direction and strength with both human-rated professionalism and with whether a question was written by an analyst or generated by an LLM. The paper claims this convergence makes question origin a valid proxy for professionalism. It also finds that professional questions tend to be shorter, more readable, and less cluttered with prefatory framing, and that a Random Forest trained only on these features reaches 0.96 accuracy in distinguishing expert from LLM questions, beating both an SVM on raw text and gemini-2.0-flash.

What carries the argument

The load-bearing instrument is a hand-annotation framework that maps each question onto four linguistically motivated categories: discourse regulators (elements like 'Thanks' or 'this one is for you' that manage the interaction), prefaces (propositions before the question that justify or contextualize it), question types (open, polar, closed-list), and request types (explanation, clarification, confirmation, data, opinion), together with readability and lexical metrics such as type-token ratio, Flesch-Kincaid, and Dale-Chall. The machinery does its work by turning the 'how' of a question into a fixed feature vector, so that correlations with human professionalism ratings and with question origin can be computed and a Random Forest can be trained on the same features. Because the features are defined in advance and are inspectable, the framework also supplies an explanation of what the classifier has learned: prefatory padding and low readability mark machine-style questions, while concise, thematically focused, readable questions read as professional.

What would settle it

Collect independent professionalism ratings for the 250 questions in the Question Origin Dataset, then recompute the Spearman correlations between features and ratings; if the correlation directions do not match the paper's Human-Rated Professionalism Dataset columns, the claim that origin is a valid proxy for professionalism fails. A second test would re-annotate a sample with two annotators and report agreement; low agreement would show the feature definitions are not stable enough to carry the result.

Watch

Extended reading notes

Core claim

The central claim is that the same set of interpretable linguistic indicators carries both human perceptions of professionalism and the distinction between expert-authored and machine-generated questions. Refined, this claim says that a question's professionalism is largely encoded in its surface-structural and pragmatic design: how it acknowledges the prior speaker, how it frames or prefaces the request, which syntactic question form it uses, what kind of answer it solicits, and how readable it is. The paper reports that these features align consistently across the Human-Rated Professionalism Dataset and the Question Origin Dataset, and interprets the alignment as evidence that origin classification can stand in for professionalism when human ratings are unavailable. A corollary finding is that clarity, not complexity, drives perceived professionalism: higher ratings go with fewer words and sentences and higher readability scores. Finally, the paper shows that a shallow classifier using only these features outperforms a large language model and a bag-of-words SVM on the origin task, which it presents as evidence that professionalism is learnable and expressible in transparent linguistic terms.

Load-bearing premise

The load-bearing premise is that the hand-annotated linguistic features were marked accurately and with the same standards in both datasets; the paper gives no annotation guidelines and no inter-annotator agreement scores, so inconsistent or noisy feature labels would undermine every correlation and the reported 0.96 classifier accuracy.

Editorial extensions

If this is right

Professionalism in financial analyst questions can be scored automatically with transparent, pre-defined linguistic features instead of expensive human annotation.
The same features give an interpretable diagnostic: a question flagged as machine-generated can be shown to be over-prefaced, low-readability, or low on discourse regulators.
Origin classification is a viable proxy for professionalism, so large unlabeled pools of expert and LLM-generated questions can be converted into training signal without collecting ratings.
Question-generation systems that aim to sound professional should reduce prefatory framing and aim for concise, readable constructions, since those are the traits associated with higher human ratings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the convergence holds beyond finance, the origin task could become a cheap labeling device for professionalism in other expert question settings such as medical or legal Q&A; the paper does not test that transfer.
A natural ablation would separate feature groups (prefaces vs. question types vs. readability) to check whether the 0.96 accuracy is carried by a single superficial cue such as word count; the paper does not report such an ablation.
The professionalism ratings may partly encode rater expectations about what an analyst should sound like, so the features' correlation with origin could be inflated by the LLM's distinct style; a test with a different LLM family or with human imitations would separate stylistic origin signals from professionalism itself.
Because the paper's Question Origin Dataset labels are binary, it treats professionalism as a category rather than a gradient; a rated version of that dataset would show whether the proxy saturates once questions are genuinely professional.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What should you know: this is a short empirical paper with a genuinely new annotated dataset and a clean feature-based classifier, but the reader's stress test lands. The central convergence claim is contradicted by the paper's own Table 3, and that contradiction is load-bearing, not cosmetic.

Credit where it is due. The Question Origin Dataset (250 human analyst questions paired against 250 LLM-generated ones) is new and appears to be available. The Random Forest result, hitting 0.96 accuracy on interpretable features and beating both a TF-IDF SVM and Gemini, is a useful, reproducible baseline for detection of machine-written questions in financial text. The feature set is grounded in prior discourse work, and the paper's limitations paragraph is honest about the single-domain scope and about origin being an imperfect proxy.

The soft spot is the center of the argument. Section 3.3 says the same linguistic indicators show consistent correlation direction and strength in both the professionalism and origin tasks. That is not what the data show. Acknowledgment is strongly positive for professionalism and strongly negative for human origin; open questions flip from strongly negative to positive; interjection count flips from strongly positive to strongly negative. The paper never acknowledges these reversals, and it never tests the proxy directly by correlating origin labels with professionalism ratings on matched items. This matters because the 0.96 accuracy could be driven by authorship-specific artifacts that have nothing to do with perceived professionalism. Also, classification results have no error bars or significance tests, and the annotation section gives no guidelines and no inter-annotator agreement, so we cannot tell whether the features were coded consistently across the two datasets.

Who is this for? Financial NLP researchers who want a cheap, transparent detector of LLM-generated analyst questions, and computational pragmatics people interested in operationalizing question style. The paper deserves a serious referee: the dataset and code make it worth engaging, and the convergence claim is testable. I would send it to review with a clear request to either test the proxy directly or substantially weaken the claim to something like "some features transfer, others do not, and here is the pattern."

Referee Report

4 major / 5 minor

Summary. The paper proposes to model professionalism in expert questioning by extracting hand-crafted linguistic features from financial analyst questions and correlating them with two targets: human-perceived professionalism ratings (HRPD) and question origin (human vs. LLM, QOD). The authors report that many features show consistent correlations with both targets, argue that question origin is therefore a valid proxy for professionalism, and train a Random Forest classifier on the origin task that reportedly reaches 0.96 accuracy, outperforming an SVM and an LLM baseline. The paper concludes that professionalism is learnable from interpretable linguistic signals.

Significance. If the convergence claim were sound, this would be a valuable contribution: it would offer an interpretable, theory-driven way to evaluate professional questioning and to detect LLM-generated content in high-stakes domains. The paper also provides a potentially reusable feature scheme and a publicly available supplementary dataset, which are constructive elements. However, the central empirical claim is contradicted by the paper's own Table 3, which shows multiple statistically significant sign reversals between the professionalism and origin correlations. Because the proxy argument rests entirely on these correlations aligning, the main thesis is not supported. The classification results are also insufficiently specified. The datasets and feature inventory may still be useful to the community, but the current analysis does not establish the paper's core conclusion.

major comments (4)

[§3.3, Table 3] The central convergence claim is directly contradicted by the paper's own data. The text states that 'many of the same linguistic indicators ... exhibit consistent correlation direction and strength in both tasks,' but Table 3 shows multiple statistically significant directional reversals: acknowledgment is ↑↑↑ in HRPD and ↓↓↓ in QOD; open question type is ↓↓↓ in HRPD and ↑ in QOD; interjection count is ↑↑↑ in HRPD and ↓↓↓ in QOD. These are not minor discrepancies but sign flips on features the paper uses to argue for a shared stylistic foundation. Since the proxy argument in §3.3 rests on alignment between the two correlation profiles, the presence of these reversals means the claim of convergence is unsupported. The paper provides no aggregate measure, such as a rank correlation across feature vectors, to quantify the extent of agreement, so the selective reading of Table 3 is insufficient.
[§3.3, §4] The proxy logic is structurally circular and is never directly tested. The paper establishes that features correlate with professionalism (HRPD) and with origin (QOD) in separate datasets, then treats origin classification accuracy as evidence that the features capture professionalism. This requires the unstated assumption that the two target variables are positively associated on matched items, which is never measured. The reported 0.96 accuracy could instead reflect LLM-specific artifacts that are unrelated or inversely related to perceived professionalism; indeed, interjection count is strongly positive for professionalism but strongly negative for human origin in Table 3. A direct test—for example, training on HRPD professionalism scores, or measuring whether predicted origin correlates with professionalism ratings on held-out items—is needed to support the paper's central claim.
[§3.1, §3.2] The feature annotation process is undocumented, making the reported correlations unverifiable. The paper does not provide annotation guidelines, annotator training, inter-annotator agreement statistics, or a description of whether features were extracted manually or automatically. Since every correlation in Table 3 and the classifier in Section 4 rely on these features, the absence of reliability evidence undermines the empirical foundation. This concern is amplified because HRPD and QOD are annotated separately, and any difference in annotation practice between the two datasets could produce spurious sign flips or false alignments.
[§4, Table 2] The classification evaluation is insufficiently specified. The paper does not report the train/test split, whether cross-validation was used, which dataset(s) the results refer to, how the 'full question set' was partitioned, or the hyperparameters of the Random Forest and SVM. Without this information, the accuracy and F1 values in Table 2 cannot be interpreted, and the comparison to gemini-2.0-flash may not be controlled (e.g., different prompting strategies, different training-data exposure, or label leakage from HRPD items that also have origin labels). The claim that a shallow feature-based model outperforms an LLM requires a fully specified evaluation protocol.

minor comments (5)

[§3.2] The caption of Table 1 says it presents all statistically significant correlations in the 'mixed and combined datasets,' but the manuscript never defines what 'mixed' and 'combined' mean; also, Table 1 is a strict subset of Table 3, yet both are captioned as reporting significant correlations, which is confusing.
[§3.2, Table 3] The feature taxonomy in the text lists four categories (discourse regulators, prefaces, question types, request types), but Table 3 also includes NLP features (type-token ratio, readability, word/sentence counts) and additional features such as filler word count, assertion count, and mean assertion length that are not described in Section 3.2.
[§4] The phrase 'full question set' is ambiguous; the authors should specify whether it consists of QOD only, HRPD only, or both, and how the origin label was obtained for HRPD items.
[§3.1, Supplementary materials] The provenance of HRPD is incompletely described: the text says it comes from Juan et al. (2025) and was shared upon request, while the supplementary link is for QOD; the authors should clarify how readers can access HRPD and confirm that its annotation procedure matches the description given.
[Tables 1 and 3] The tables report Spearman correlations but no sample sizes are shown; the authors should report N for each correlation to allow readers to assess the precision of the estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proxy-validation argument rests on an empirical convergence claim that is contradicted by the paper's own Table 3, but that is a correctness risk, not a circular reduction of a prediction to its inputs.

full rationale

After walking the derivation chain, I find no step in which a prediction is equivalent by construction to an input, or in which a fitted parameter is renamed as a prediction. The feature set (Section 3.2) is theory-driven and defined independently of the two target variables; the correlations in Table 1/3 are computed on data (HRPD from Juan et al. 2025, QOD constructed here) and are not used to fit features. The Random Forest classifier (Section 4) is trained on origin labels—not on professionalism ratings—so its 0.96 accuracy is a genuine out-of-sample result on the origin task, even though the section title overstates it as 'predicting professionalism.' The only load-bearing external input is HRPD, which comes from a co-authored prior paper; however, it is a shared dataset, not an unverified uniqueness theorem or ansatz, so it does not constitute circularity. The paper's own Limitations section explicitly acknowledges that origin is a proxy and cannot fully capture professionalism. The convergence claim in Section 3.3 is contradicted by several entries in Table 3 (e.g., acknowledgment and interjection count are positive in HRPD and negative in QOD; open questions are negative in HRPD and positive in QOD), but an internal empirical inconsistency is a correctness risk, not a circular reduction. Accordingly, steps are empty and score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its free parameters are undisclosed design choices (RF hyperparameters, LLM generation settings). The axioms are domain assumptions about the validity of the professionalism ratings, the proxy, dataset comparability, and annotation reliability.

free parameters (2)

Random Forest hyperparameters = not reported
Accuracy of 0.96 is reported without specifying tree count, depth, split criteria, or validation protocol; the result may be sensitive to undisclosed tuning choices.
LLM generation settings for QOD machine questions = not reported
The LLM-written questions are not described in terms of model, prompt, temperature, or post-processing; this choice directly affects how easy the origin classification is.

assumptions (4)

domain assumption Professionalism is a valid construct measurable by crowd-sourced 3-point ratings
The HRPD uses mean ratings from five annotators on a 3-point scale (Section 3.1); the paper treats these as ground truth for professionalism without validating the construct.
domain assumption Question origin is a valid proxy for professionalism
Section 3 states origin can serve as a proxy, but the paper does not directly show that origin predicts professionalism ratings; instead it infers the proxy from overlapping correlations.
domain assumption The two datasets (HRPD and QOD) are comparable and drawn from the same population
HRPD and QOD are separate samples (Section 3.1) with different selection procedures; comparing correlations across them assumes comparability.
domain assumption The linguistic features are annotated reliably and consistently
Section 3.2 describes the feature categories but gives no annotation guidelines or inter-annotator agreement, so reliability is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Modeling Professionalism in Expert Questioning through Linguistic Differentiation." pith.science (2026). https://pith.science/paper/EO33Z76O

@misc{pith2026250720249,
  author       = {Pith},
  title        = {Pith review of: Modeling Professionalism in Expert Questioning through Linguistic Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EO33Z76O}},
  note         = {Machine review of arXiv:2507.20249}
}

read the original abstract

Professionalism is a crucial yet underexplored dimension of expert communication, particularly in high-stakes domains like finance. This paper investigates how linguistic features can be leveraged to model and evaluate professionalism in expert questioning. We introduce a novel annotation framework to quantify structural and pragmatic elements in financial analyst questions, such as discourse regulators, prefaces, and request types. Using both human-authored and large language model (LLM)-generated questions, we construct two datasets: one annotated for perceived professionalism and one labeled by question origin. We show that the same linguistic features correlate strongly with both human judgments and authorship origin, suggesting a shared stylistic foundation. Furthermore, a classifier trained solely on these interpretable features outperforms gemini-2.0 and SVM baselines in distinguishing expert-authored questions. Our findings demonstrate that professionalism is a learnable, domain-general construct that can be captured through linguistically grounded modeling.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Tony Berber Sardinha. 2024. Ai-generated vs human-authored texts: A multidimensional comparison. Applied Corpus Linguistics, 4(1):100083

work page 2024
[4]

Clayman and John Heritage

Steven E. Clayman and John Heritage. 2002. The News Interview: Journalists and Public Figures on the Air. Cambridge University Press

work page 2002
[5]

Clayman, John Heritage, Marc N

Steven E. Clayman, John Heritage, Marc N. Elliott, and Laurie L. McDonald. 2007. When does the watchdog bark? conditions of aggressive questioning in presidential news conferences. American Sociological Review, 72(1):23--41

work page 2007
[6]

Belinda Crawford Camiciottoli. 2010. Earnings calls: Exploring an emerging financial reporting genre. Discourse & Communication, 4(4):343--359

work page 2010
[7]

Giulia D'Agostino, Andrea Rocci, and Chris Reed. 2024. https://aclanthology.org/2024.finnlp-2.10 Capturing Analysts ' Questioning Strategies in Earnings Calls via a Question Cornering Score ( QCS ) . In Proceedings of the Eighth Financial Technology and Natural Language Processing and the 1st Agent AI for Scenario Planning , pages 107--118, Jeju, South Korea

work page 2024
[8]

de Oliveira and S \'i lvia M

Maria L. de Oliveira and S \'i lvia M. R. Pereira. 2018. Formulations in delicate actions: A study of analyst questions in earnings conference calls. International Journal of Business Communication, 55(3):293--309

work page 2018

Show all 14 references

[9]

Paul Drew and John Heritage. 1992. Analyzing talk at work: an introduction. In Paul Drew and John Heritage, editors, Talk at Work: Interaction in Institutional Settings, pages 3--65. Cambridge University Press

1992
[10]

Cornelia Ilie. 2021. Questions we (inter)act with: Interrelatedness of questions and answers in discourse. In Cornelia Ilie, editor, Questioning and Answering Practices across Contexts and Cultures, pages 1--32. John Benjamins

2021
[11]

Yining Juan, Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2025. Co-trained retriever-generator framework for question generation in earnings calls. In Companion Proceedings of the ACM Web Conference 2025

2025
[12]

Rudi Palmieri, Andrea Rocci, and Nadzeya Kudrautsava. 2015. https://doi.org/10.1016/j.scoms.2015.03.014 Argumentation in earnings conference calls. Corporate standpoints and analysts' challenges . Studies in communication sciences, 15, 2015(1):120--132

2015 doi
[13]

Srikant Sarangi and Celia Roberts. 1999. Talk, Work and Institutional Order: Discourse in Medical, Mediation and Management Settings. Mouton de Gruyter, Berlin

1999
[14]

Zanotto and Segun Aroyehun

Silvia E. Zanotto and Segun Aroyehun. 2024. Human variability vs. machine consistency: A linguistic analysis of texts generated by humans and large language models. https://arxiv.org/abs/2412.03025. ArXiv:2412.03025

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Tony Berber Sardinha. 2024. Ai-generated vs human-authored texts: A multidimensional comparison. Applied Corpus Linguistics, 4(1):100083

work page 2024

[4] [4]

Clayman and John Heritage

Steven E. Clayman and John Heritage. 2002. The News Interview: Journalists and Public Figures on the Air. Cambridge University Press

work page 2002

[5] [5]

Clayman, John Heritage, Marc N

Steven E. Clayman, John Heritage, Marc N. Elliott, and Laurie L. McDonald. 2007. When does the watchdog bark? conditions of aggressive questioning in presidential news conferences. American Sociological Review, 72(1):23--41

work page 2007

[6] [6]

Belinda Crawford Camiciottoli. 2010. Earnings calls: Exploring an emerging financial reporting genre. Discourse & Communication, 4(4):343--359

work page 2010

[7] [7]

Giulia D'Agostino, Andrea Rocci, and Chris Reed. 2024. https://aclanthology.org/2024.finnlp-2.10 Capturing Analysts ' Questioning Strategies in Earnings Calls via a Question Cornering Score ( QCS ) . In Proceedings of the Eighth Financial Technology and Natural Language Processing and the 1st Agent AI for Scenario Planning , pages 107--118, Jeju, South Korea

work page 2024

[8] [8]

de Oliveira and S \'i lvia M

Maria L. de Oliveira and S \'i lvia M. R. Pereira. 2018. Formulations in delicate actions: A study of analyst questions in earnings conference calls. International Journal of Business Communication, 55(3):293--309

work page 2018