Pith. sign in

REVIEW 4 major objections 7 minor 16 references

Dialogue-Based Multi-Dimensional Relationship Extraction from Novels

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes CREDI, a dialogue-aware method that extracts character relationships from novels on three parallel dimensions at once, and reports that it outperforms both traditional baselines and existing state-of-the-art systems on…

desk verdict New dataset and a simple dialogue-construction trick are the real value; the public-dataset SOTA comparison is under-specified enough to withhold judgment on. read the letter →

arxiv 2507.04852 v1 pith:2ZJ6NJXC submitted 2025-07-07 cs.CL

Yuchen Yan , Hanjie Zhao , Senbin Zhu , Hongde Liu , Zhihong Zhang , Yuxiang Jia This is my paper · ORCID
classification cs.CL
keywords characterrelationshipextractiondialoguerelationnoveltextunderstandinglargelanguagemodelsLoRAfine-tuningin-contextlearningChinesedatasetliteraryanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that character relationship extraction from novels is best treated as a dialogue-understanding problem rather than a flat text-classification task. To show this, the authors build CREDI, which rebuilds novel quotations into explicit 'A said to B' dialogue units, separates relationships into the three dimensions of polarity, type, and generational hierarchy, and fine-tunes a Llama 3.1 model with LoRA plus retrieval-based in-context learning. They also construct a new Chinese dataset from a classic martial-arts novel, with 10,773 relationship labels, and report weighted F1 scores of 0.79, 0.76, and 0.74 on the three dimensions, beating their baselines and prior systems. If the claim holds, it offers a practical route to automated social-network construction from literary texts and a reusable benchmark for the novel domain.

What carries the argument

The central mechanism is the Expanded Dialogue construction: the model converts narrative quotations into explicit speaker-to-addressee exchanges in the format 'A said to B,' which makes interaction structure visible to the LLM. Around this, CREDI wraps relationship dimension separation (three parallel coarse labels instead of a single fine-grained label), structured prompts encoding target characters and candidate labels, retrieval-based in-context learning using multilingual BERT embeddings with FAISS Top-K retrieval, and LoRA low-rank fine-tuning of the Query and Value matrices of Llama 3.1.

What would settle it

Measure inter-annotator agreement on a held-out sample of the NCRE annotations; if agreement is low, or if CREDI's advantage over GPT-3.5 few-shot and other baselines shrinks substantially when the labels are re-annotated and corrected, the reported superiority would be an artifact of label noise rather than a genuine modeling gain.

Watch

Extended reading notes

Core claim

CREDI's central claim is that the implicit, context-heavy relationships between characters in novels become tractable when the input is restructured into dialogue form and when the label space is organized into three coarse parallel dimensions. The paper shows that a LoRA fine-tuned Llama 3.1 model, prompted with expanded dialogues that name both speaker and addressee, outperforms BERT classification, fine-tuned T5, and GPT-3.5 zero- and few-shot prompting on the new NCRE dataset, and also surpasses published SOTA on FiRe, CRECIL, and DialogRE with weighted F1 scores of 0.71, 0.63, and 0.80 respectively. The ablation study attributes part of the gain to the expanded-dialogue construction and part to multi-dimensional joint learning, and a case study demonstrates the model capturing hostile emotion, a master-apprentice tie, and senior-junior hierarchy in one passage.

Load-bearing premise

The NCRE gold labels, produced by two rounds of annotation with no reported inter-annotator agreement statistic, are assumed accurate and consistent enough to serve as ground truth for both fine-tuning and evaluation.

Editorial extensions

If this is right

  • Automated character relationship networks for novels can be built at scale, with edge colors and thickness reflecting extracted polarity and dialogue frequency.
  • The dialogue-restructuring strategy may transfer to other narrative genres, such as historical fiction or translated literature, where implicit relationships are common.
  • The NCRE dataset provides a new benchmark for Chinese literary relation extraction with a multi-dimensional label schema.
  • Separating relationship polarity, type, and generational hierarchy could support finer-grained literary analysis than single-label relation extraction.
  • The reported performance gains suggest that parameter-efficient fine-tuning plus dialogue restructuring is a viable alternative to large-scale prompt engineering for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the method's advantage persists on novels with less dialogue-dense prose, such as heavily descriptive or modernist narratives; a test on such texts would clarify the boundary of the dialogue-restructuring benefit.
  • The asymmetric example (Huang Rong versus Ouyang Ke) hints that directional dialogue construction could support directional or evolving relationship extraction, but the paper does not exploit this explicitly.
  • The claimed superiority over SOTA depends on the NCRE annotation quality; without an inter-annotator agreement measure, the head-to-head improvements could partly reflect label noise, a risk the reader should weigh before building on the numbers.
  • A natural next step the paper does not take is to model relationship change over narrative time, using the dialogue positions to track when a polarity or hierarchy label shifts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces CREDI, a relation-extraction method for Chinese novels built on LoRA fine-tuned Llama 3.1, with three main components: a three-dimensional label schema (polarity, type, generational hierarchy), an 'expanded dialogue' construction that adds speaker-listener structure, and retrieval-based in-context learning. It also presents NCRE, a newly annotated Chinese dataset derived from Jin Yong's The Legend of the Condor Heroes, with 10,773 relation labels. Experiments compare CREDI against fine-tuned T5, BERT classification, and GPT-3.5 zero-shot/few-shot on NCRE, and against baselines and an unattributed 'SOTA' row on FiRe, CRECIL, and DialogRE. The reported weighted F1 scores are 0.79/0.76/0.74 on NCRE and 0.63/0.71/0.80 on the public datasets; the paper claims these results demonstrate consistent superiority. The final section illustrates an application to building character relationship networks for four novels.

Significance. If the claims hold, the paper makes two useful contributions: a moderately sized Chinese literary relation-extraction dataset with a multi-dimensional label schema, and an empirical recipe showing that dialogue restructuring and LoRA fine-tuning improve LLM performance on character relation extraction. The dataset fills a real gap, since existing dialogue-relation corpora are drawn from screenplays rather than novels, and the relationship-network visualization is a sensible application. The strongest parts of the paper are the concrete dataset construction and the positive results of the dialogue-expansion ablation. However, the significance is currently capped by three load-bearing empirical gaps: the absence of inter-annotator reliability evidence for NCRE, the absence of variance or significance measures for any of the reported F1 values, and the unverifiable comparison against SOTA on public benchmarks due to unspecified data modification, label mapping, and metric provenance. These are fixable in a revision, but without them the central comparative claims are not yet established.

major comments (4)
  1. [Section 4.2, Table 4] The central claim that CREDI outperforms existing SOTA on FiRe, CRECIL, and DialogRE is not verifiable from the information given. The text states that for CRECIL and DialogRE 'label threshold filtering and random sampling strategies are applied to balance the data and reduce dataset size,' but it never says whether these operations are restricted to the training split or also alter the test split. If the test split is modified, the published SOTA numbers from the original benchmarks are not comparable to CREDI's scores. In addition, the 'SOTA' row in Table 4 is unattributed: no citation is provided for the values 0.62, 0.38, and 0.79, so their provenance (split, metric, label set) cannot be checked. The unusually large FiRe gap (SOTA 0.38 vs. CREDI 0.71) makes this concern concrete rather than hypothetical.
  2. [Section 3.1] No inter-annotator agreement measure is reported for the NCRE corpus. The annotation process is described as a two-round iterative labeling and review procedure, but the paper provides no kappa/alpha statistic, no count of annotators, and no account of how disagreements were adjudicated. Since all of Tables 2 and 3 are evaluated against this ground truth, the reliability of the 10,773 labels is load-bearing for the paper's central empirical claims. If annotation noise is high, the reported F1 differences between CREDI and the baselines may be artifacts of label inconsistency rather than genuine method superiority.
  3. [Section 4.3, Tables 2-4] All experimental results are reported as single F1 values with no standard deviations across random seeds, no confidence intervals, and no significance tests. This is particularly concerning on NCRE, where the 8:1:1 split of 1,109 dialogue units leaves a test set of roughly 110 units, making the observed differences between CREDI and the baselines—especially the more modest gaps in the ablation study in Table 3—potentially consistent with sampling noise. The paper should report results across multiple seeds and provide significance tests or bootstrap confidence intervals.
  4. [Section 4.2, Table 4] The label-schema mapping between CREDI's three coarse dimensions and the fine-grained labels of FiRe (12 types), DialogRE (36 types), and CRECIL (30 types) is not described. If CREDI is evaluated on a coarsened or converted label set while the SOTA numbers are based on the original fine-grained labels, the comparison is invalid even if the splits are identical. The paper must state, for each public dataset, the exact label mapping, the evaluation metric, and whether the original test split was used without any filtering or subsampling.
minor comments (7)
  1. [Section 4.2] The sentence 'To prevent knowledge leakage, character names in the DIR2E training set are randomly replaced with unique codes' refers to an undefined dataset name 'DIR2E'; this appears to be a typo or an unexplained abbreviation, and the passage should be corrected or clarified.
  2. [Section 4.3] The baseline labeled 'Fine-tuned PTMs' in Table 2 is never explicitly defined; Section 4.2 describes a T5-based fine-tuned generative model, but the table label should match the baseline description for readability.
  3. [Section 3.2, Figure 3] The text states that 'node size is determined by the smoothed frequency of direct quotations,' but the smoothing parameter is never specified; either report the parameter value or remove the word 'smoothed'.
  4. [Section 4.4] The case study example in Table 5 will be difficult for many readers to verify because the Chinese passage is reproduced with colored highlighting that may not survive typesetting; adding a plain-text transcription or an English gloss would improve accessibility.
  5. [Section 4.5] There is a typo in 'TheThe Legend of the Condor Heroes' in the first paragraph of Section 4.5; it should read 'The Legend of the Condor Heroes'.
  6. [Section 2, References] The sentence containing reference [6] has a dangling bracket: '[...] and has become a widely used benchmark [16]. [6] In recent work, Li et al. [8]...' The '[6]' should be removed or integrated into a proper citation.
  7. [Section 4.1] The paper does not report the LoRA rank, LoRA alpha, FAISS top-k retrieval count, learning rate, or number of training epochs; these hyperparameters are necessary for reproducibility and should be listed in an appendix or experimental settings section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CREDI is evaluated on held-out splits of newly constructed and public datasets with standard supervised train/test separation.

full rationale

This is an empirical supervised-learning paper, not a derivation. CREDI's inputs (Llama 3.1, LoRA fine-tuning, expanded dialogue reconstruction, dimension-separated prompts, FAISS-based retrieval) are not defined in terms of the target labels, and the headline numbers in Tables 2 and 4 are held-out weighted-F1 scores on an 8:1:1 split for NCRE and the original splits for CRECIL and DialogRE. There is no fitted parameter that is renamed as a prediction, no equation equating an output to an input, and no load-bearing self-citation chain: the cited prior work (DialogRE, CRECIL, CoPrompt, PRIDE, LoRA, Llama 3.1) is external, and the paper does not invoke any uniqueness theorem from its own authors. The reviewer-visible concerns are real but are not circularity: the unattributed SOTA row and the label-threshold filtering/random-sampling applied to CRECIL and DialogRE raise comparability risks, and the absence of inter-annotator agreement in Section 3.1 raises label-quality risk. Those are correctness and validity concerns, not circularity by construction, because the evaluation protocol still separates training from testing and does not reduce the reported predictions to the training labels. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger is empty of invented entities. The free parameters are design choices that are not reported or tuned, and the axioms are the domain assumptions that underpin the empirical pipeline.

free parameters (4)
  • FAISS top-k retrieval count
    Retrieval-based in-context learning requires specifying how many similar examples to include in the prompt; the paper does not report k or any sweep over it.
  • LoRA rank and alpha
    Parameter-efficient fine-tuning targets Query and Value matrices, but the rank and scaling parameters are not reported.
  • Label threshold and sampling ratio for CRECIL and DialogRE balancing
    The authors filter and resample these datasets to reduce imbalance, but the exact thresholds and sample sizes are not given, making the evaluation protocol under-specified.
  • Smoothing parameter for node-size visualization
    The relationship network uses 'smoothed frequency' of quotes, but the smoothing method is not defined.
assumptions (5)
  • domain assumption Annotator labels are reliable and consistent
    The paper uses the NCRE labels as ground truth for fine-tuning and evaluation, but reports no inter-annotator agreement (Section 3.1).
  • domain assumption Expanded 'A said to B' dialogue format preserves the narrative context needed for relation extraction
    The method converts narrative into dialogue form; the paper does not analyze cases where this transformation loses background information that may be necessary for labels (Section 4.1).
  • ad hoc to paper The three relationship dimensions are sufficient and well-separated for the target domain
    The polarity/type/generation schema is introduced for Jin Yong's martial arts fiction and is not derived from a theory of narrative relationships (Section 3.1).
  • domain assumption Fine-tuning Llama 3.1 on a small corpus generalizes to other novels and to English dialogue datasets
    The method transfers to FiRe (English) and DialogRE (Chinese/English) despite training on Chinese novel dialogue, with no analysis of cross-domain degradation (Section 4.3).
  • domain assumption Multilingual BERT embeddings are adequate for similarity retrieval of Chinese novel dialogue
    Retrieval uses multilingual BERT without comparison to other embedding choices, and retrieval quality is not evaluated separately (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dialogue-Based Multi-Dimensional Relationship Extraction from Novels." pith.science (2026). https://pith.science/paper/2ZJ6NJXC

@misc{pith2026250704852,
  author       = {Pith},
  title        = {Pith review of: Dialogue-Based Multi-Dimensional Relationship Extraction from Novels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZJ6NJXC}},
  note         = {Machine review of arXiv:2507.04852}
}
read the original abstract

Relation extraction is a crucial task in natural language processing, with broad applications in knowledge graph construction and literary analysis. However, the complex context and implicit expressions in novel texts pose significant challenges for automatic character relationship extraction. This study focuses on relation extraction in the novel domain and proposes a method based on Large Language Models (LLMs). By incorporating relationship dimension separation, dialogue data construction, and contextual learning strategies, the proposed method enhances extraction performance. Leveraging dialogue structure information, it improves the model's ability to understand implicit relationships and demonstrates strong adaptability in complex contexts. Additionally, we construct a high-quality Chinese novel relation extraction dataset to address the lack of labeled resources and support future research. Experimental results show that our method outperforms traditional baselines across multiple evaluation metrics and successfully facilitates the automated construction of character relationship networks in novels.

Figures

Figures reproduced from arXiv: 2507.04852 by the authors.

Figure 1
Figure 1. Character Relationship Extraction from Novels The main contributions of this paper are summarized as follows: • A multi-dimensional character relationship extraction framework is proposed, which recognizes relationship types based on dialogue structure and incorpo￾rates a LLMs-based task optimization strategy, improve model adaptability to complex novel texts. • A high-quality Chinese novel character relationship ex… view at source ↗
Figure 2
Figure 2. Annotation Process of Character Relationship Data The core objective of the annotation process was to determine the relation￾ship between two specified characters through contextual understanding of the surrounding paragraph. For each instance, annotators selected the most appro￾priate relationship label across three parallel dimensions. These dimensions pro￾vide a multi-layered framework for relationship analysis, … view at source ↗
Figure 3
Figure 3. Friendship-Hostility Relationship Network of Characters in Novel The character relationship network reveals the overall structure and features of interpersonal relationships in the novel, as shown in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Character Relationship Extraction with Dialogue Information Relationship Category Division To address issues of fine-grained and over￾lapping relationship labels in traditional methods, CREDI reorganizes relation￾ships into three parallel dimensions with coarse-grained…
Figure 5
Figure 5. Figure 5: Character Relationship Network in Heaven Sword and Dragon Saber 5 Conclusion This paper aims to achieve automated extraction of multi-dimensional character relationships in novels. To extend the scope of relation extraction to literary texts, we construct a Chinese cor…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 12 canonical work pages

  1. [1]

    Intelli- gent Systems with Applications19, 200244 (2023)

    Detroja, K., Bhensdadia, C., Bhatt, B.S.: A survey on relation extraction. Intelli- gent Systems with Applications19, 200244 (2023)

    work page 2023

  2. [2]

    arXiv preprint arXiv:2407.21783 (2024)

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  3. [3]

    ICLR1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)

    2022

  4. [4]

    Jiang, Y., Xu, Y., Zhan, Y., He, W., Wang, Y., Xi, Z., Wang, M., Li, X., Li, Y., Yu, Y.: The crecil corpus: a new dataset for extraction of relations between characters in chinese multi-party dialogues (2022)

    work page 2022

  5. [5]

    IEEE Transactions on Big Data7(3), 535–547 (2019)

    Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with gpus. IEEE Transactions on Big Data7(3), 535–547 (2019)

    2019

  6. [6]

    Applied Sciences13(22), 12414 (2023)

    Kim, J., Kim, G., Son, J., Lim, H.: Prompt language learner with trigger generation for dialogue relation extraction. Applied Sciences13(22), 12414 (2023)

    work page 2023

  7. [7]

    ACM Computing Surveys (CSUR)52(5), 1–40 (2019)

    Labatut, V., Bost, X.: Extraction and analysis of fictional character networks: A survey. ACM Computing Surveys (CSUR)52(5), 1–40 (2019)

    work page 2019

  8. [8]

    Empirical Analysis of Dialogue Relation Extraction with Large Language Models

    Li, G., Xu, Z., Shang, Z., Liu, J., Ji, K., Guo, Y.: Empirical analysis of dialogue relation extraction with large language models. arXiv preprint arXiv:2404.17802 (2024)

    work page Pith review arXiv 2024

Show all 16 references
  1. [9]

    In: 2019 IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS)

    Li,J.,Zhang,C.,Tan,H.,Li,C.:Complexnetworksofcharactersinfictionalnovels. In: 2019 IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS). pp. 417–420. IEEE (2019)

    2019

  2. [10]

    Li, X., Polat, F., Groth, P., et al.: Do instruction-tuned large language models help with relation extraction? (2023)

    2023

  3. [11]

    In: Proceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval

    Li, Y., Jiang, Y., Chen, J., Wang, L., Tao, Y., Zhang, Y.: Coprompt: A contrast- prompt tuning method for multiparty dialogue character relationship extraction. In: Proceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval. p...

    2023

  4. [12]

    Springer (1988)

    Page, N.: Speech in the English novel. Springer (1988)

    1988

  5. [13]

    Advances in Complex Systems19(03), 1650005 (2016)

    Prado, S.D., Dahmen, S.R., Bazzan, A.L., Carron, P.M., Kenna, R.: Temporal network analysis of literary texts. Advances in Complex Systems19(03), 1650005 (2016)

    2016

  6. [14]

    In: The Conference on Empirical Methods in Natural Language Processing

    Tigunova, A., Mirza, P., Yates, A., Weikum, G.: Pride: Predicting relationships in conversations. In: The Conference on Empirical Methods in Natural Language Processing. pp. 4636–4650. ACL (2021)

    2021

  7. [15]

    arXiv preprint arXiv:2004.08056 (2020)

    Yu, D., Sun, K., Cardie, C., Yu, D.: Dialogue-based relation extraction. arXiv preprint arXiv:2004.08056 (2020)

    2020 arXiv

  8. [16]

    Neurocomputing456, 299–311 (2021)

    Zhao, L., Xu, W., Gao, S., Guo, J.: Utilizing graph neural networks to improving dialogue-based relation extraction. Neurocomputing456, 299–311 (2021)

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.