Pith. sign in

REVIEW 4 major objections 4 minor 80 references

A pipeline that mines the full text of scientific preprints and patent applications with large language models identifies retrieval-augmented generation and conversational agents as the two emerging transformative technologies as of 2024.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 07:31 UTC pith:JRXI77TT

load-bearing objection A genuinely large and competently assembled pipeline, but the emergence-detection claim is under-validated: the Jaccard signal is computed over triples, not papers, and the RAG–conversational-agent finding is a retrospective hit with no baselines or significance tests. the 4 major comments →

arxiv 2510.25370 v2 pith:JRXI77TT submitted 2025-10-29 cs.CL

Monitoring Transformative Technological Convergence Through LLM-Extracted Semantic Entity Triple Graphs

Alexander Sternfeld , Andrei Kucharavy , Dimitri Percia David , Alain Mermoud , Julian Jang-Jaccard , Nathan Monnet This is my paper
classification cs.CL
keywords technology forecastingtechnological convergencesemantic tripleslarge language modelsnoun staplingJaccard similarityknowledge graphspatent analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to turn technology forecasting from expert guesswork into a measurable signal. It builds a huge graph of technology-related facts by having language models read the full text of scientific preprints and patent applications and extract subject–predicate–object triples — statements like 'retrieval-augmented generation improves factual consistency.' The pipeline groups similar technology terms into topics (a step they call noun stapling), then watches how often pairs of topics appear in the same triples over time, using Jaccard similarity as the convergence meter. The claim is that a sustained rise in that similarity flags technologies that are converging and becoming transformative. Applied to 278,625 preprints and 9,793 patent applications, the method flags retrieval-augmented generation and conversational agents as the two emerging transformative technologies as of 2024.

Core claim

The paper's central discovery is a pipeline that can identify emerging transformative technologies at scale from raw full text alone, without curated ontologies or expert panels. Language models extract semantic triples, a string-similarity step groups near-synonymous technology terms into topics, and a time series of Jaccard similarity between topic pairs reveals convergence: a rise in the ratio of shared triples to total triples. On two corpora — scientific preprints and US patent applications — the pipeline reproduces known LLM-era trends and, as of 2024, identifies retrieval-augmented generation and conversational agents as the two technologies whose convergence with the broader LLM fiel

What carries the argument

Semantic triple graph: each sentence is reduced to (subject, predicate, object) triples by an LLM, connecting technology-designating nouns. Noun stapling: a soft-Dice string similarity measure that groups terms like 'retrieval augmented' and 'retrieval augmentation' into one topic, so naming variation does not hide merges. Jaccard similarity time series: for each pair of topics, the ratio of triples shared by both to triples appearing in either is computed per time slice; the paper's load-bearing rule is that a rise in this ratio over time marks technological convergence. The pipeline also uses community detection to find established clusters and term-frequency trend decomposition to separat

Load-bearing premise

The load-bearing premise is that an increase in Jaccard similarity between two topics' triple sets reflects true technological convergence rather than artifacts of publishing volume, naming overlap, or LLM hallucination — and this link is asserted in the paper without being tested against a known set of historically convergent technologies.

What would settle it

Run the pipeline on texts from a past era with a well-documented convergence — for example, the melt of neural networks into speech recognition during the 2010s — and check whether the Jaccard curve for those two topics rises at the historically observed time. If it stays flat or rises only after the merger is obvious in hindsight, the proposed signal fails as an early indicator.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Technology forecasters can run the pipeline on any large text corpus — news, blogs, reports — without expert-curated taxonomies, and get early convergence warnings.
  • The two named outputs, retrieval-augmented generation and conversational agents, are presented as concrete candidates for transformative technologies as of 2024, testable against subsequent developments.
  • Because the method works on patents as well as preprints, it can connect early scientific signals to downstream commercial adoption timelines.
  • The scale of triple extraction (23.8 million triples) sets a new baseline for full-text bibliometric analysis, beyond metadata- and citation-based methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the Jaccard signal is validated against historical convergences, the same pipeline could be run retrospectively on decades of literature to build a verified library of convergence curves, turning this from a case study into a forecasting instrument.
  • The method's dependence on LLM-extracted triples means hallucinated or redundant triples could inflate co-occurrence; a natural extension is to weight triples by extraction confidence or to filter through factuality checks before computing similarity.
  • The Jaccard metric treats all triples equally and does not normalize for corpus growth, so a spurious rise after a hype event like a major product release is possible; a normalized or background-subtracted version could make the signal more robust to such shocks.
  • The paper leaves open whether the two flagged technologies will bear out; if they do, the method offers a template for identifying future convergences in other ICT subfields.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a full-text pipeline for technology forecasting. It extracts semantic triples from 278,625 arXiv preprints and 9,793 USPTO patent applications using a few-shot-prompted Llama-3-8B model, after benchmarking spaCy and several LLMs on a hand-labeled dataset built from a survey of LLMs. The triples are post-processed, filtered, grouped into technology topics via a string-similarity 'noun stapling' method, and analyzed with Louvain clustering, edge trend classification, and Jaccard-similarity time series. The paper's central claim is that the pipeline can identify both established and emerging convergence patterns, and specifically that retrieval-augmented generation and conversational agents were emerging transformative technologies as of end 2024.

Significance. If valid, the contribution is significant: it moves bibliometric forecasting from title/abstract metadata to full text, at a scale (about 23.8 million triples) beyond typical co-word studies, and it makes a concrete, falsifiable forecast. The pipeline is carefully engineered, the extraction benchmark is transparent, and the data/code repository is a strength. The main weakness is that the convergence signal itself is not validated: the interpretation of Eq. (6) as technology convergence is asserted rather than tested, and no baseline or null model is provided. The headline findings are therefore plausible but not yet supported by sufficient evidence.

major comments (4)
  1. [Section 4.8, Eq. (6) / Section 5.2.4] The text in §5.2.4 describes Figure 7 as the 'topic co-publication frequency,' but Eq. (6) is computed on sets of triples, not sets of papers. Table 5 shows that the chosen LLM extracts up to 15.3 triples per paragraph, so one broad survey paper in the 2023–24 LLM corpus can place many triples into the intersection of two topic sets and inflate J(A,B). The post-ChatGPT surge in exactly such high-yield survey/position papers is thus an alternative, non-convergence explanation for the observed Jaccard increases. A document-level Jaccard (paper as the unit) or a null model that preserves per-paper triple counts and topic margins is required before the RAG–conversational-agent finding can be accepted.
  2. [Section 4.8 / Figure 7] The paper asserts that 'a technology convergence will be identifiable by an increase in the Jaccard similarity' but provides no validation against known technological convergences, no comparison with the co-word [17] or LDA [13] baselines cited in Section 2, and no significance tests or confidence intervals. Since Figure 7 displays the ten pairs with the largest observed increase, the upward trends are guaranteed by the selection rule; the informative quantity is whether those increases exceed what a null model or a baseline method would produce. The early years (2018–2021) have very small triple counts, making yearly Jaccard estimates high-variance; without uncertainty bounds the 2018 baseline is not interpretable.
  3. [Section 4.6, Eq. (5)] The noun-stapling threshold (0.85) is the mechanism that maps surface terms to topics, and every downstream analysis inherits its errors. The threshold is justified only by the assertion that 'one can be relatively certain' above it; no precision/recall evaluation against a labeled term-similarity set and no comparison with embedding-based grouping are reported. Given that topic labels such as 'Retrieval Augmented' are themselves the output of this clustering, the stability of the Figure 7 findings should be checked under reasonable threshold variations, e.g. 0.80–0.90.
  4. [Section 3.1 / Section 5.1 / Conclusion] The two component validations are both drawn from inside the target domain: the triple-extraction golden set comes from a single LLM survey paper [74], and the abbreviation benchmark comes from a co-authored cyber-defense report [47]. This is not formal circularity, but it means the model choice and all empirical cutoffs (Sections 4.4–4.6) are tuned to LLM/security discourse. The abstract and conclusion claim a 'scalable and generalizable framework' across domains and data sources; that claim is not supported by the current held-out evidence. A hold-out domain (e.g., a non-LLM technology area in the same arXiv categories) should be added.
minor comments (4)
  1. [Throughout] Typos include 'Entrepeneurship,' 'prepring(s),' 'te USPTO,' 'semantinc graph analysis,' and 'rawn text.' A careful proofreading pass is needed.
  2. [Table 2] The text says three abbreviation detection algorithms are compared, but the table lists four (Schwartz-Hearst, scispaCy, NLPRe, and fine-tuned RoBERTa).
  3. [Table 6] The caption says 20 topics but the table lists 21 rows.
  4. [Figure 7] The caption contains the duplicated phrase 'largest increase between in the Jaccard similarity,' and the plot appears to omit axis labels and units, making the yearly scale hard to interpret.

Circularity Check

2 steps flagged

Convergence is operationalized as Jaccard increase and the triple-extraction benchmark is drawn from the LLM-survey domain being forecast, so the headline 'emerging transformative technologies' is partly the selection metric renamed, with no independent validation.

specific steps
  1. self definitional [§4.8 Eq. (6); §5.2.4, Fig. 7; §6]
    "Let A(t) and B(t) be the sets of triplets that are related to topics A and B at time t, respectively. Then, the Jaccard similarity between the topics A and B at time t is defined as ... A technology convergence will be identifiable by an increase in the Jaccard similarity. ... we can designate retrieval augmentation and conversational agents as emerging transformative technologies in the field of LLMs, as of end 2024, based on research preprints published on arXiv."

    The 'emerging transformative technology' finding is produced by taking the topic pairs with the largest increase in Eq. (6) and re-labeling them as convergence/transformative. Because the preceding sentence defines a technology convergence as an increase in that same Jaccard similarity, the reported result is the selection criterion restated: the top-increase pair is designated as convergence by construction. No independent ground truth (known convergences, expert forecasts, downstream impact, or a null model) is used to confirm that Jaccard increase means transformation. The paper itself states 'We leave to future research the task of validating the identified transformative technologies,' so the forecast content of the headline result is the Eq. (6) metric itself, renamed.

  2. fitted input called prediction [§3.1; §4.1; §5.2.1; §6]
    "we manually construct a training dataset based on the paper A Survey of Large Language Models [74]. We chose this paper given our focus on the LLMs transformative technology, since it is a comprehensive and general review of LLM component technologies. ... we extract key terms from those papers related to LLMs ... we were able to identify two emerging transformative technologies as of 2024: retrieval-augmented generation and conversational agents."

    The LLM used for triple extraction is few-shot/fine-tuned and evaluated on a golden dataset built from a survey of LLM component technologies, which is the same domain from which the case study then 'identifies' retrieval-augmented generation and conversational agents. The prompt examples and the survey text supply the LLM-related vocabulary used downstream, so the pipeline is tuned on the very domain it is asked to forecast. This does not mechanically force the time trend, but it means the claimed discovery is seeded by the benchmark input, and the 'scalable and generalizable framework' claim is not tested outside the tuning domain. This closes part of the validation loop rather than providing an independent test.

full rationale

The pipeline's formal derivation is largely self-contained: text is preprocessed, triples are extracted, terms are stapled and clustered, and Eq. (6) computes a well-defined triple-set Jaccard statistic. No equation literally fits a parameter and then re-predicts that parameter, and the triple-extraction comparison does include a manually labeled 20% holdout, so not everything is circular. However, the central forecasting claim is construct-circular in two related ways. First, 'technology convergence' is defined as an increase in Eq. (6), and the paper's headline result is obtained by taking the pairs with the largest such increase and calling them 'emerging transformative technologies'; no external benchmark validates that the metric corresponds to genuine transformation. Second, the only extraction benchmark and the few-shot/fine-tuning examples are drawn from an LLM survey, i.e., from the target domain, so the vocabulary that generates the RAG/conversational-agent finding is partly an input rather than an independent discovery. The abbreviation-detection benchmark uses a co-author's report (ref [47]), but this choice is not load-bearing for the convergence analysis, so it does not raise the score further. Overall, the result has independent computational content, but the 'prediction' label is attached to the metric's output without external validation, giving a partial circularity score of 4.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The central claim rests on a chain of assumptions that are asserted rather than measured: the Jaccard-based convergence construct, the reliability of 20M+ LLM-extracted triples benchmarked on only 547 hand-labeled triples from one survey paper, the semantic adequacy of string-similarity grouping at threshold 0.85, the sufficiency of one arXiv category subset plus seed-term screening to define the field, and the premise that arXiv leads patents. Numerous cutoffs were set 'empirically' with no sensitivity analysis. No new physical or mechanistic entities are postulated; the discovered 'topics' are clusters of the pipeline's own outputs.

free parameters (9)
  • noun stapling similarity threshold = 0.85
    Chosen 'to be conservative' (§4.6); no sensitivity analysis; determines which terms get grouped into topics, directly shaping the topic graph.
  • bookcorpus score cutoff = top 10%
    'cutoff determined empirically' (§4.5); controls which terms are kept as technology-designating rather than generic.
  • frequency filter threshold = f_cc,t < 5 removed
    Control-corpus frequency floor (§4.5); set by hand to discard rare nouns.
  • subject/object length cutoff = 6 words
    'chosen empirically, based on manual inspection of triple nouns' (§4.4).
  • minimum word length = 3 characters
    'chosen empirically' (§4.4); affects which terms survive post-processing.
  • Louvain resolution = 0.85
    Gephi implementation with resolution 0.85 (§4.8); sets community granularity of the topic network.
  • KeyBERT terms per abstract = 10
    Maximum 10 key terms extracted per abstract (§4.7); caps the keyword pool feeding topic clustering.
  • edge color thresholds = 70% occurrences in 2022+ (red) / 70% in 2021− (blue)
    Drives the 'emerging vs waning' narrative in Figures 5, 6, and 9 (§5.2); arbitrary cutoffs with no sensitivity analysis.
  • topic pairs displayed in Figure 7 = 10 pairs with largest Jaccard increase
    Post-hoc selection of the most favorable pairs shapes the convergence narrative (§5.2.4); selection after seeing the data.
axioms (6)
  • domain assumption An increase in Jaccard similarity between topics' triple sets signals genuine technological convergence
    Asserted in §4.8 ('A technology convergence will be identifiable by an increase in the Jaccard similarity'); never validated against ground truth; plausibly confounded by the corpus-wide LLM publication surge and common-source papers spreading triples across many topics (Fig. 7).
  • domain assumption LLM-extracted triples are factually reliable at scale
    Triple quality is benchmarked only on the 547-triple golden dataset from one survey paper (§3.1, §5.1, Table 5), then applied to 278k papers / 20.8M triples without a second quality check; artifacts in Table 6 ('question answerin', 'art deep', 'chatgpt1') indicate scale noise.
  • ad hoc to paper String similarity at threshold 0.85 groups semantically similar technology terms
    Noun stapling uses SoftDICE character/token similarity (Eqs. 3–5) with a hand-set threshold; semantic equivalence of merged strings is assumed, not measured (§4.6).
  • domain assumption The arXiv category subset plus LLM seed terms adequately delimits the field of interest
    Corpus = cs.CL, cs.LG, cs.AI, cs.IR, stat.ML 2018–2024, with papers screened by 'large language model'/'llm' in title/abstract (§3.2, §4.7); excludes adjacent ML fields, and the 'generalizable framework' claim rests on a single-field demonstration.
  • domain assumption arXiv preprints lead patent applications as early technology signals
    Stated as motivation (§3.3) and used to interpret the arXiv-to-patent lag in §5.3.1; not tested as a hypothesis.
  • domain assumption The author-annotated golden triples are ground truth for extraction quality
    Table 5 benchmarks against 547 manually annotated triples; these annotations themselves show 7.2% inconsistency at Levenshtein distance 2 (§5.1), and are produced by the authors from a single domain survey (§3.1).

pith-pipeline@v1.3.0-alltime-deepseek · 22441 in / 22357 out tokens · 207710 ms · 2026-08-04T07:31:24.005986+00:00 · methodology

0 comments
read the original abstract

Forecasting transformative technologies remains a critical but challenging task, particularly in fast-evolving domains such as Information and Communication Technologies (ICTs). Traditional expert-based methods struggle to keep pace with short innovation cycles and ambiguous early-stage terminology. In this work, we propose a novel, data-driven pipeline to monitor the emergence of transformative technologies by identifying patterns of technological convergence. Our approach leverages advances in Large Language Models (LLMs) to extract semantic triples from unstructured text and construct a large-scale graph of technology-related entities and relations. We introduce a new method for grouping semantically similar technology terms (noun stapling) and develop graph-based metrics to detect convergence signals. The pipeline includes multi-stage filtering, domain-specific keyword clustering, and a temporal trend analysis of topic co-occurence. We validate our methodology on two complementary datasets: 278,625 arXiv preprints (2017--2024) to capture early scientific signals, and 9,793 USPTO patent applications (2018-2024) to track downstream commercial developments. Our results demonstrate that the proposed pipeline can identify both established and emerging convergence patterns, offering a scalable and generalizable framework for technology forecasting grounded in full-text analysis.

Figures

Figures reproduced from arXiv: 2510.25370 by Alain Mermoud, Alexander Sternfeld, Andrei Kucharavy, Dimitri Percia David, Julian Jang-Jaccard, Nathan Monnet.

Figure 1
Figure 1. Figure 1: The complete pipeline for the triple extraction and downstream analysis, starting from either raw patent data or raw arXiv data. The green components of the pipeline reflect the triple extraction procedure, including the pre- and post-processing steps. The grey components of the pipeline illustrate the key-term extraction, noun stapling and the downstream analyses. triple has a subject or object of which 𝑓… view at source ↗
Figure 2
Figure 2. Figure 2: The number of papers for each of the 15 most common topics in the field of LLMs, based on the extracted and grouped key terms. Each of the bars is divided into the arXiv categories from which the papers originate. the field is the simplified interaction between humans and LLMs. For a more fine-grained analysis, we investigate the trends of individual key terms within the topics. Again, we distinguish betwe… view at source ↗
Figure 3
Figure 3. Figure 3: Multiple line plot showing for each of the 10 most common topics the number of papers in which at least one triple appears for that topic. The gray dashed line shows the aggregate seasonal component, whereas the colored lines show the trend for each topic. between retrieval augmented and conversational agent is stronger and is not yet stagnating. In combination with the network analysis that highlighted th… view at source ↗
Figure 4
Figure 4. Figure 4: The trends for key sub-technology terms for the 10 most common emerging technology topics in the LLM space, based on the extracted and grouped key terms. The gray dashed line shows the aggregate seasonal component that the key terms share, whereas the colored lines show the trend for each key term. machine learning methods, corresponding to the trend of LLM-based code generation and analysis integration in… view at source ↗
Figure 5
Figure 5. Figure 5: Clusters of topics composed using the Louvain method for community detection. Relations in red occur over 70% of the time in 2022 or later, whereas relations in blue occur over 70% of the time in 2021 or earlier. All other relations are in black. Node/edge sizes correspond to absolute frequency, and text size to eigenvector network centrality. Sternfeld, A. et al.: Preprint submitted to Elsevier Page 14 of… view at source ↗
Figure 6
Figure 6. Figure 6: Emerging (red), disappearing (blue), and persistent (black) relations between technology-related term clusters based on the arXiv paper data. Relations in red occur over 70% of the time in 2022 or later, whereas relations in blue occur over 70% of the time in 2021 or earlier. All other relations are displayed as black. The node color reflects the term cluster they belong to, the sizes of the nodes and edge… view at source ↗
Figure 7
Figure 7. Figure 7: The Jaccard similarities for the 10 pairs of topics for which the Jaccard similarity had the largest increase between in the Jaccard similarity between 2018 and 2024. We consider all topics, as identified by the extracted and grouped key terms in for the topic of LLMs [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Multiple line plot showing for each of the 10 most common topics, as identified by the extracted and grouped key terms, the number of patent applications in which at least one triple appears for that topic. scale impossible. Specifically, we (I) leveraged the recent advances in NLP and recently published benchmarking datasets for to extract information relevant to technologi￾cal forecasting in an unbiased … view at source ↗
Figure 9
Figure 9. Figure 9: Network representation of the key topics based on the USPTO patent application data. Relations in red are occurring over 70% of the time in 2022 or later, whereas relations in blue occur over 70% of the time in 2021 or earlier. All other relations are displayed as black. The node color corresponds to the technology term cluster they belong to, the sizes of the nodes and edges correspond to the absolute fre… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 11 canonical work pages

  1. [1]

    arXiv preprint server

    , . arXiv preprint server. https://info.arxiv.org. Accessed: 2025-04-28

    2025

  2. [2]

    biorXiv preprint server

    , . biorXiv preprint server. https://www.biorxiv.org/ content/about-biorxiv. Accessed: 2025-04-28

    2025

  3. [3]

    medXiv preprint server

    , . medXiv preprint server. https://www.medrxiv.org/ content/about-medrxiv. Accessed: 2025-04-28

    2025

  4. [4]

    PubMed Central online publications archive

    , . PubMed Central online publications archive. https://pmc. ncbi.nlm.nih.gov/about/intro/. Accessed: 2025-04-28

    2025

  5. [5]

    Technology forecasting: A case study of computational technologies

    Adamuthe, A.C., Thampi, G.T., 2019. Technology forecasting: A case study of computational technologies. Technological Forecasting and Social Change 143, 181–189. URL: https://www.sciencedirect.com/science/ article/pii/S0040162518302890, doi: https: //doi.org/10.1016/j.techfore.2019.03.002

    work page doi:10.1016/j.techfore.2019.03.002 2019

  6. [6]

    Llama 3 model card URL: https://github

    AI@Meta, 2024. Llama 3 model card URL: https://github. com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  7. [7]

    Artifex, 2024. Pymupdf. https://pypi.org/project/ PyMuPDF/. Accessed: 2024-02-29

    2024

  8. [8]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y ., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N.,...

    Pith/arXiv arXiv 2022

  9. [9]

    Gephi: An open source software for exploring and manipulating net- works

    Bastian, M., Heymann, S., Jacomy, M., 2009. Gephi: An open source software for exploring and manipulating net- works. URL: http://www.aaai.org/ocs/index.php/ ICWSM/09/paper/view/154

    2009

  10. [10]

    Bastos, A., Nadgeri, A., Singh, K., Mulang, I.O., Shekarpour, S., Hof- fart, J., Kaul, M., 2021. Recon: Relation extraction using knowledge graph context in a graph neural network, in: Proceedings of the Web Conference 2021, Association for Computing Machinery, New York, NY , USA. p. 1673–1685. URL: https://doi.org/10.1145/ 3442381.3449917, doi:10.1145/34...

    arXiv 2021

  11. [11]

    Scibert: Pretrained language model for scientific text, in: EMNLP

    Beltagy, I., Lo, K., Cohan, A., 2019. Scibert: Pretrained language model for scientific text, in: EMNLP. arXiv:arXiv:1903.10676

    Pith/arXiv arXiv 2019

  12. [12]

    The semantic web: A new form of web content that is meaningful to computers will unleash a revolution of new possibilities

    Berners-Lee, T., Hendler, J., Lassila, O., 2001. The semantic web: A new form of web content that is meaningful to computers will unleash a revolution of new possibilities. ScientificAmerican.com

    2001

  13. [13]

    Latent dirichlet allocation

    Blei, D.M., Ng, A.Y ., Jordan, M.I., 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022

    2003

  14. [14]

    Fast unfolding of communities in large networks

    Blondel, V .D., Guillaume, J.L., Lambiotte, R., Lefebvre, E., 2008. Fast unfolding of communities in large networks. Journal of statisti- cal mechanics: theory and experiment 2008, P10008. Sternfeld, A. et al.: Preprint submitted to Elsevier Page 17 of 21 Monitoring Transformative Technological Convergence

    2008

  15. [15]

    Language models are few-shot learners, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A...

    2020

  16. [16]

    Technology Forecasting: Recent Trends and New Methods

    Calleja-Sanz, G., Nadal, J., Solé-Parellada, F., 2020. Technology Forecasting: Recent Trends and New Methods. pp. 45–69. doi: 10. 1007/978-3-030-40896-1_3

    2020

  17. [17]

    From translations to problematic networks: An introduction to co-word analysis

    Callon, M., Courtial, J.P., Laville, F., 1983. From translations to problematic networks: An introduction to co-word analysis. Social Science Information 22, 191–235

    1983

  18. [18]

    Massive open index of scholarly papers launches

    Chawla, D.S., 2022. Massive open index of scholarly papers launches. Nature URL: https://api.semanticscholar. org/CorpusID:246278314

    2022

  19. [19]

    The innovator’s dilemma: When new technologies cause great firms to fail

    Christensen, C.M., 1997. The innovator’s dilemma: When new technologies cause great firms to fail. Harvard Business School Press, Boston

    1997

  20. [20]

    Cohan, A., Feldman, S., Beltagy, I., Downey, D., Weld, D.S.,

  21. [21]

    Persistent Forecasting of Disruptive Technologies" Report 2

    Council, N.R., on Engineering, D., Sciences, P., on Forecasting Future Disruptive Technologies, C., 2010. Persistent Forecasting of Disruptive Technologies" Report 2. National Academies Press

    2010

  22. [22]

    Digital transformations: new tools and methods for mining technological intelligence

    Daim, T., Yalçin, H., 2022. Digital transformations: new tools and methods for mining technological intelligence. Edward Elgar Publishing

    2022

  23. [23]

    Anticipat- ing future innovation pathways through large data analysis

    Daim, T.U., Chiavetta, D., Porter, A.L., Saritas, O., 2016. Anticipat- ing future innovation pathways through large data analysis. Springer

    2016

  24. [24]

    An experimental study of group opinion: the delphi method

    Dalkey, N., 1969. An experimental study of group opinion: the delphi method. Futures 1, 408–426

    1969

  25. [25]

    PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts, in: Kondrak, G., Watanabe, T

    Dernoncourt, F., Lee, J.Y ., 2017. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts, in: Kondrak, G., Watanabe, T. (Eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (V olume 2: Short Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan. pp. 308–313. URL...

    2017

  26. [26]

    Technological paradigms and technological trajec- tories: A suggested interpretation of the determinants and directions of technical change

    Dosi, G., 1982. Technological paradigms and technological trajec- tories: A suggested interpretation of the determinants and directions of technical change. Research Policy 11, 147–162. URL: https: //api.semanticscholar.org/CorpusID:16840352

    1982

  27. [27]

    Identifying potentially disruptive trends by means of keyword network analysis

    Dotsika, F., Watkins, A., 2017. Identifying potentially disruptive trends by means of keyword network analysis. Technological Forecasting and Social Change 119, 114–127. URL: https://www.sciencedirect.com/science/ article/pii/S0040162517303517, doi: https: //doi.org/10.1016/j.techfore.2017.03.020

    work page doi:10.1016/j.techfore.2017.03.020 2017

  28. [28]

    Span-based joint entity and rela- tion extraction with transformer pre-training, in: Giacomo, G.D., Catalá, A., Dilkina, B., Milano, M., Barro, S., Bugarín, A., Lang, J

    Eberts, M., Ulges, A., 2020. Span-based joint entity and rela- tion extraction with transformer pre-training, in: Giacomo, G.D., Catalá, A., Dilkina, B., Milano, M., Barro, S., Bugarín, A., Lang, J. (Eds.), ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2...

    2020

  29. [29]

    Organization strategy and structural differences for radical versus incremental innovation

    Ettlie, J.E., Bridges, W.P., O’keefe, R.D., 1984. Organization strategy and structural differences for radical versus incremental innovation. Management Science 30, 682–695. URL: https: //api.semanticscholar.org/CorpusID:154030756

    1984

  30. [30]

    Improving generalization in large langue model by learning prefix subspaces, in: Bouamor, H., Pino, J., Bali, K

    Falissard, L., Guigue, V ., Soulier, L., 2023. Improving generalization in large langue model by learning prefix subspaces, in: Bouamor, H., Pino, J., Bali, K. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore. pp. 11474–11483. URL: https://aclanthology.org/2023.findings-emn...

    work page doi:10.18653/v1/2023.findings-emnlp.768 2023

  31. [31]

    Argumentation mining in scientific literature for sustainable development, in: Al-Khatib, K., Hou, Y ., Stede, M

    Fergadis, A., Pappas, D., Karamolegkou, A., Papageorgiou, H., 2021. Argumentation mining in scientific literature for sustainable development, in: Al-Khatib, K., Hou, Y ., Stede, M. (Eds.), Proceedings of the 8th Workshop on Argument Mining, Association for Computational Linguistics, Punta Cana, Dominican Republic. pp. 100–111. URL: https://aclanthology.o...

    work page doi:10.18653/v1/2021.argmining-1.10 2021

  32. [32]

    A simple substitution model of technological change

    Fisher, J.C., Pry, R.H., 1971. A simple substitution model of technological change. Technological forecasting and social change 3, 75–88

    1971

  33. [33]

    Gali, N., Mariescu-Istodor, R., Hostettler, D., Fränti, P.,

  34. [34]

    Keybert: Minimal keyword extraction with bert

    Grootendorst, M., 2020. Keybert: Minimal keyword extraction with bert. URL: https://doi.org/10.5281/zenodo. 4461265, doi:10.5281/zenodo.4461265

    doi:10.5281/zenodo 2020

  35. [35]

    Technology futures analysis: Toward integration of the field and new methods

    Group, T.F.A.M.W., et al., 2004. Technology futures analysis: Toward integration of the field and new methods. Technological Forecasting and Social Change 71, 287–303

    2004

  36. [36]

    Forecasting technology emergence using large-scale knowledge graphs and link prediction

    Gu, M., Liu, X., Zhang, L., Wang, J., 2024. Forecasting technology emergence using large-scale knowledge graphs and link prediction. Scientometrics In press

    2024

  37. [37]

    Knowledge graphs

    Hogan, A., Blomqvist, E., Cochez, M., D’amato, C., Melo, G.D., Gutierrez, C., Kirrane, S., Gayo, J.E.L., Navigli, R., Neumaier, S., Ngomo, A.C.N., Polleres, A., Rashid, S.M., Rula, A., Schmelzeisen, L., Sequeda, J., Staab, S., Zimmermann, A., 2021. Knowledge graphs. ACM Computing Surveys 54, 1–37. URL: http://dx. doi.org/10.1145/3447772, doi:10.1145/3447772

    doi:10.1145/3447772 2021

  38. [38]

    spaCy: Industrial-strength Natural Language Processing in Python doi:10.5281/zenodo.1212303

    Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A., 2020. spaCy: Industrial-strength Natural Language Processing in Python doi:10.5281/zenodo.1212303

    doi:10.5281/zenodo.1212303 2020

  39. [39]

    Lora: Low-rank adaptation of large language models

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W., 2021. Lora: Low-rank adaptation of large language models. arXiv:2106.09685

    Pith/arXiv arXiv 2021

  40. [40]

    Extracting core claims from scientific articles

    Jansen, T., Kuhn, T., 2017. Extracting core claims from scientific articles. CoRR abs/1707.07678. URL: http://arxiv.org/ abs/1707.07678, arXiv:1707.07678

    Pith/arXiv arXiv 2017

  41. [41]

    Technological forecasting in perspective

    Jantsch, E., 1967. Technological forecasting in perspective. Number 21,931 in OECD Publications, OECD, Paris

    1967

  42. [42]

    Mistral 7b

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E., 2023. Mistral 7b. arXiv:2310.06825

    Pith/arXiv arXiv 2023

  43. [43]

    A novel approach to forecast promising technology through patent analysis

    Kim, G., Bae, J., 2017. A novel approach to forecast promising technology through patent analysis. Technological Forecasting and Social Change 117, 228–237. URL: https://www.sciencedirect.com/science/ article/pii/S0040162516307661, doi: https: //doi.org/10.1016/j.techfore.2016.11.023

    work page doi:10.1016/j.techfore.2016.11.023 2017

  44. [44]

    Technology fusion and the new r&d

    Kodama, F., 1992. Technology fusion and the new r&d. Harvard business review , 70–78

    1992

  45. [45]

    Forecasting re- search trends using dynamic graph representations: The science4cast 2023 challenge

    Krenn, M., Bollen, J., Stojanovski, J., et al., 2023. Forecasting re- search trends using dynamic graph representations: The science4cast 2023 challenge. arXiv preprint arXiv:2301.03589 . Sternfeld, A. et al.: Preprint submitted to Elsevier Page 18 of 21 Monitoring Transformative Technological Convergence

    Pith/arXiv arXiv 2023

  46. [46]

    From deep neural language models to llms, in: Large Language Models in Cybersecurity: Threats, Exposure and Mitigation

    Kucharavy, A., 2024. From deep neural language models to llms, in: Large Language Models in Cybersecurity: Threats, Exposure and Mitigation. Springer, pp. 3–17

    2024

  47. [47]

    Fundamentals of generative large language models and perspectives in cyber-defense

    Kucharavy, A., Schillaci, Z., Maréchal, L., Würsch, M., Dolamic, L., Sabonnadiere, R., David, D.P., Mermoud, A., Lenders, V ., 2023. Fundamentals of generative large language models and perspectives in cyber-defense. arXiv:2303.12132

    Pith/arXiv arXiv 2023

  48. [48]

    Scientific discourse tagging for evidence extraction, in: Merlo, P., Tiedemann, J., Tsarfaty, R

    Li, X., Burns, G.A., Peng, N., 2021. Scientific discourse tagging for evidence extraction, in: Merlo, P., Tiedemann, J., Tsarfaty, R. (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V olume, EACL 2021, Online, April 19 - 23, 2021, Association for Compu- tational Linguistics. pp. 255...

    work page doi:10.18653/v1/2021.eacl-main.218 2021

  49. [49]

    Monitoring and forecasting the development trends of nanogenerator technology using citation analysis and text mining

    Li, X., Fan, M., Zhou, Y ., Fu, J., Yuan, F., Huang, L., 2020. Monitoring and forecasting the development trends of nanogenerator technology using citation analysis and text mining. Nano Energy 71, 104636. URL: https://www.sciencedirect. com/science/article/pii/S2211285520301932, doi:https://doi.org/10.1016/j.nanoen.2020. 104636

    work page doi:10.1016/j.nanoen.2020 2020

  50. [50]

    A novel integrated approach for quantifying the convergence of disruptive technologies from science to technol- ogy

    Li, X., Wang, Y ., 2024. A novel integrated approach for quantifying the convergence of disruptive technologies from science to technol- ogy. Technological Forecasting and Social Change URL: https: //api.semanticscholar.org/CorpusID:273569421

    2024

  51. [51]

    Liu, L., Omidvar, A., Ma, Z., Agrawal, A., An, A., 2022. Un- supervised knowledge graph generation using semantic similar- ity matching, in: Cherry, C., Fan, A., Foster, G., Haffari, G.R., Khadivi, S., Peng, N.V ., Ren, X., Shareghi, E., Swayamdipta, S. (Eds.), Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing...

    2022

  52. [52]

    A survey of link prediction in complex networks

    Martínez, V ., Berzal, F., Cubero, J.C., 2016. A survey of link prediction in complex networks. ACM Computing Surveys (CSUR) 49, 1–33

    2016

  53. [53]

    context2vec: Learning generic context embedding with bidirectional LSTM, in: Riezler, S., Goldberg, Y

    Melamud, O., Goldberger, J., Dagan, I., 2016. context2vec: Learning generic context embedding with bidirectional LSTM, in: Riezler, S., Goldberg, Y . (Eds.), Proceedings of the 20th SIGNLL Con- ference on Computational Natural Language Learning, Associa- tion for Computational Linguistics, Berlin, Germany. pp. 51–61. URL: https://aclanthology.org/K16-1006...

    2016

  54. [54]

    Quantitative analysis of culture using millions of digitized books

    Michel, J.B., Shen, Y .K., Aiden, A.P., Veres, A., Gray, M.K., Team, G.B., Pickett, J.P., Hoiberg, D., Clancy, D., Norvig, P., et al., 2011. Quantitative analysis of culture using millions of digitized books. science 331, 176–182

    2011

  55. [55]

    Distributed representations of words and phrases and their composi- tionality, in: Neural Information Processing Systems

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., 2013. Distributed representations of words and phrases and their composi- tionality, in: Neural Information Processing Systems. URL:https: //api.semanticscholar.org/CorpusID:16447573

    2013

  56. [56]

    ScispaCy: Fast and robust models for biomedical natural language processing, in: Demner-Fushman, D., Cohen, K.B., Ananiadou, S., Tsujii, J

    Neumann, M., King, D., Beltagy, I., Ammar, W., 2019. ScispaCy: Fast and robust models for biomedical natural language processing, in: Demner-Fushman, D., Cohen, K.B., Ananiadou, S., Tsujii, J. (Eds.), Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy. pp. 319–327. URL: https://aclanthology....

    doi:10.18653/v1/w19-5034 2019

  57. [57]

    Triplet extraction leveraging sentence transformers and dependency parsing

    Ottersen, S.G., Pinheiro, F., Bação, F., 2024. Triplet extraction leveraging sentence transformers and dependency parsing. Array 21, 100334. URL: https://www.sciencedirect. com/science/article/pii/S2590005623000590, doi:https://doi.org/10.1016/j.array.2023. 100334

    work page doi:10.1016/j.array.2023 2024

  58. [58]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al., 2022. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems 35, 27730–27744

    2022

  59. [59]

    Measuring security development in information technologies: A scientometric framework using arxiv e-prints

    Percia David, D., Maréchal, L., Lacube, W., Gillard, S., Tsesmelis, M., Maillart, T., Mermoud, A., 2023. Measuring security development in information technologies: A scientometric framework using arxiv e-prints. Technological Forecasting and Social Change 188, 122316. URL: https://www.sciencedirect. com/science/article/pii/S004016252300001X, doi:https://...

    work page doi:10.1016/j.techfore.2023 2023

  60. [60]

    Technology forecasting based on semantic and citation analysis of patents: A case of robotics domain

    Qiu, Z., Wang, Z., 2022. Technology forecasting based on semantic and citation analysis of patents: A case of robotics domain. IEEE Transactions on Engineering Management 69, 1216–1236. doi: 10. 1109/TEM.2020.2978849

    arXiv 2022

  61. [61]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al., 2018. Improving language understanding by generative pre-training

    2018

  62. [62]

    Economic theory and entrepreneurial his- tory, in: Clemence, R

    Schumpeter, J.A., 1949. Economic theory and entrepreneurial his- tory, in: Clemence, R. (Ed.), Essays on Entrepreneurs, Innovations, Business Cycles, and the Evolution of Capitalism. New Jersey: Transaction Publishers, pp. 272–286

    1949

  63. [63]

    A simple algorithm for identifying abbreviation definitions in biomedical text, in: Altman, R.B., Dunker, A.K., Hunter, L., Klein, T.E

    Schwartz, A.S., Hearst, M.A., 2003. A simple algorithm for identifying abbreviation definitions in biomedical text, in: Altman, R.B., Dunker, A.K., Hunter, L., Klein, T.E. (Eds.), Proceedings of the 8th Pacific Symposium on Biocomputing, PSB 2003, Lihue, Hawaii, USA, January 3-7, 2003, pp. 451–

    2003

  64. [64]

    Citation analysis

    Smith, L.C., 1981. Citation analysis

    1981

  65. [65]

    Llm-resilient bibliometrics: Factual consistency through entity triplet extraction, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y

    Sternfeld, A., Kucharavy, A., David, D.P., Mermoud, A., Jang- Jaccard, J., 2024. Llm-resilient bibliometrics: Factual consistency through entity triplet extraction, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y . (Eds.), Proceed- ings of Joint Workshop of the 5th Extraction and Evaluation of Knowledge Entities from Scientifi...

    2024

  66. [66]

    Travis Hoppe, H.B., 2024. Nlpre. https://github.com/ NIHOPA/NLPre. Accessed: 2024-04-15

    2024

  67. [67]

    A knowledge-based information extraction prototype for data-rich documents in the information technology domain

    Vargas, S.G.J., 2008. A knowledge-based information extraction prototype for data-rich documents in the information technology domain. Diss. National University of Columbia, Bogota

    2008

  68. [68]

    New trends in bibliometric apis: A com- parative analysis

    Velez-Estevez, A., Pérez, I.J., García-Sánchez, P., Moral-Munoz, J.A., Cobo, M.J., 2023. New trends in bibliometric apis: A com- parative analysis. Inf. Process. Manag. 60, 103385. URL: https: //api.semanticscholar.org/CorpusID:258730412

    2023

  69. [69]

    Early identification of breakthrough technologies: Insights from science-driven innovations

    Wang, D., Zhou, X., Zhao, P., Pang, J., Ren, Q., 2025. Early identification of breakthrough technologies: Insights from science-driven innovations. Journal of Informetrics 19, 101606. URL: https://www.sciencedirect. com/science/article/pii/S1751157724001184, doi:https://doi.org/10.1016/j.joi.2024.101606

    arXiv 2025

  70. [70]

    Claimdistiller: Scientific claim extraction with supervised contrastive learning, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y

    Wei, X., Hoque, M.R.U., Wu, J., Li, J., 2023. Claimdistiller: Scientific claim extraction with supervised contrastive learning, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y . (Eds.), Proceedings of Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE2023) and the 3rd AI +...

    2023

  71. [71]

    Llm- based entity extraction is not for cybersecurity, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y

    Würsch, M., Kucharavy, A., David, D.P., Mermoud, A., 2023. Llm- based entity extraction is not for cybersecurity, in: Zhang, C., Zhang, Y ., Mayr, P., Lu, W., Suominen, A., Chen, H., Ding, Y . (Eds.), Proceedings of Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE2023) and the 3rd AI + Informetrics ...

    2023

  72. [72]

    Llms perform poorly at concept extraction in cyber-security research liter- ature

    Würsch, M., Kucharavy, A., David, D.P., Mermoud, A., 2023. Llms perform poorly at concept extraction in cyber-security research liter- ature. arXiv:2312.07110

    Pith/arXiv arXiv 2023

  73. [73]

    Development trend forecasting for coherent light generator technology based on patent citation network anal- ysis

    You, H., Li, M., Hipel, K.W., Jiang, J., Ge, B., Duan, H., 2017. Development trend forecasting for coherent light generator technology based on patent citation network anal- ysis. Scientometrics 111, 297–315. URL: https:// doi.org/10.1007/s11192-017-2252-y , doi: 10.1007/ S11192-017-2252-Y

    work page doi:10.1007/s11192-017-2252-y 2017

  74. [74]

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., Du, Y ., Yang, C., Chen, Y ., Chen, Z., Jiang, J., Ren, R., Li, Y ., Tang, X., Liu, Z., Liu, P., Nie, J.Y ., Wen, J.R.,

  75. [75]

    Starling-7b: Improving llm helpfulness & harmlessness with rlaif

    Zhu, B., Frick, E., Wu, T., Zhu, H., Ganesan, K., Chiang, W.L., Zhang, J., Jiao, J., 2023. Starling-7b: Improving llm helpfulness & harmlessness with rlaif

    2023

  76. [76]

    Zilio, L., Saadany, H., Sharma, P., Kanojia, D., Or ˘asan, C., 2022. PLOD: An abbreviation detection dataset for scientific documents, in: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., Piperidis, S. (Eds.), Proceedings of the Thirteenth Language Re- sour...

    2022

  77. [462]

    URL: http://psb.stanford.edu/psb-online/ proceedings/psb03/schwartz.pdf

  78. [2019]

    Expert Systems with Applications 129, 169–185

    Framework for syntactic string similarity measures. Expert Systems with Applications 129, 169–185. URL: https://www.sciencedirect.com/science/ article/pii/S0957417419302222, doi: https: //doi.org/10.1016/j.eswa.2019.03.048

    work page doi:10.1016/j.eswa.2019.03.048 2019

  79. [2020]

    URL: https://arxiv.org/ abs/2004.07180, arXiv:2004.07180

    Specter: Document-level representation learning using citation-informed transformers. URL: https://arxiv.org/ abs/2004.07180, arXiv:2004.07180

    Pith/arXiv arXiv 2004

  80. [2023]

    arXiv:2303.18223

    A survey of large language models. arXiv:2303.18223

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.