{"id":"3829f74f-3f62-4520-89dd-ab3a7507532c","arxiv_id":"2412.01093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A regex-plus-GPT-4 pipeline extracts acronym-expansion pairs from scientific papers more fully than either component alone, though precision is not measured.","lead":"This paper combines a regular expression parser with GPT-4 to find acronyms and their full forms in scientific PDFs. The hybrid beats either tool alone on a 200-paper test set, but accuracy is measured only for expansions that the authors' own parser found.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II counts GPT-4's non-empty outputs as correct expansions without validation; the claimed superiority of GPT+RegEx+Pre is therefore unestablished.","rationale":"I agree with the reader's weakest_assumption. The paper's strongest claim appears in Sections III-C and VII with Table II reporting that GPT+RegEx+Pre expands 72-85% of regex-detected acronyms, versus 15-21% for regex alone. The load-bearing condition for that claim is that every GPT-4 output counted in the numerator is a correct expansion and that the regex-detected acronym set is a fair denominator. Neither is established. The paper's own Section VI states that parser accuracy statistics could not be collected and that the manual review only verified pattern-conforming acronym recall; it also acknowledges the parser misses lowercase/number-led acronyms. Meanwhile, the prompt explicitly permits GPT-4 to guess when the full form is not determinable, so Table II's 'Expansions Found' likely measures non-empty responses rather than verified correct expansions. The proposed concrete test, measuring precision and recall on a sampled set with independent verification, would settle whether the hybrid's advantage is real or an artifact. Since the reader already assigned a CONDITIONAL verdict for exactly this reason, I do not recommend changing the verdict.","tokens_in":13034,"tokens_out":2944,"duration_ms":29590,"concrete_test":"Sample 50 papers (or at least 200 acronyms) from the combined GPT+RegEx+Pre output across the four domains. Independently verify each returned expansion against the source paper text and, where the expansion is absent from the text, against a domain reference such as a web search or an authoritative dictionary, with two annotators. Compute precision and recall against manually annotated acronym-expansion pairs for those papers. If precision is materially below the reported 'expansions found' rate, or if recall against the true acronym set is much lower, the reported advantage of the combined method is an artifact of unverified GPT outputs. Report per-domain numbers and GPT-4 run-to-run variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sections III-C and VII, Table II) that GPT+RegEx+Pre is 'best' rests on two unvalidated quantities. First, the denominator: 'Total Acronyms' is whatever the regex parser flags after preprocessing (Listing 1), not a gold-standard set of acronyms in the paper. The authors explicitly state that they could not collect statistics regarding the parser's accuracy (Section VI) and that their 40-paper manual check only confirmed that the parser finds all acronyms matching its own pattern. This admits both false positives (chemical names, headings) and false negatives (lowercase- or number-led acronyms, acknowledged in Section VII). Second, the numerator: 'Expansions Found' in Table II counts GPT-4 outputs as correct without checking them. The prompt (Listing 4) instructs GPT-4 to 'use your best judgment' when the full form cannot be determined, so non-empty hallucinated answers are counted as successful expansions. The paper reports only three cases where GPT-4 failed to produce an expansion; it reports no cases where an expansion was produced but was wrong. Since GPT alone is shown at '100%' of its self-selected acronym set, and RegEx alone finds only about 20%, the entire advantage of the hybrid over RegEx could be an artifact of accepting LLM guesses as ground truth. Without precision measurement on GPT-4 outputs, the conclusion that the integrated approach 'yields the best results' outruns the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid pipeline for extracting acronym-expansion pairs from scientific PDFs, combining document preprocessing, a regular-expression-based parser, and GPT-4 (specifically GPT-4o mini) prompts. The parser identifies acronyms and extracts expansions that follow parenthetical patterns; GPT-4 is then used to verify, correct, or infer expansions for acronyms whose expansions were not found. The method is evaluated on 200 arXiv papers from four domains. The central claim, stated in Sections III-C and VII, is that the integrated approach (GPT+RegEx+Pre) outperforms both the regex parser alone and GPT-4 alone, with Table II reporting expansion-finding rates of 72.2% to 84.9% for the hybrid versus roughly 15% to 21% for the regex parser alone.","tokens_in":13455,"tokens_out":2478,"duration_ms":23307,"significance":"If the performance claims were supported by a rigorous evaluation, the paper would offer a practical and potentially useful engineering contribution: combining regex-based identification with an LLM for expansion inference in a token-efficient, domain-agnostic way. The paper also documents concrete preprocessing challenges for PDF-derived text. However, the current evidence does not substantiate the headline claim. The evaluation in Table II counts GPT-4 outputs as correct without any gold-standard labels or precision measurement, and the denominator of acronyms is defined by the authors' own regex parser rather than by an independently validated set. The paper explicitly acknowledges in Section VI that no parser accuracy statistics were collected. As a result, the claimed advantage of the hybrid method over the regex parser alone is not established. The work is reproducible only in a loose sense: the regex patterns and prompt are given, but the dataset, code, and raw outputs are not provided.","major_comments":[{"comment":"The numerator 'Expansions Found' counts any non-empty GPT-4 output as a correct expansion without validation. The prompt in Listing 4 explicitly instructs GPT-4 to 'use your best judgment to provide the most accurate and concise full form' when the full form cannot be determined from context. This means hallucinated or plausible-but-wrong expansions are counted as successes. The paper reports only three cases where GPT-4 failed to produce any expansion and no cases where an produced expansion was wrong. Without a precision measurement on a sample of GPT-4 outputs, the reported 72.2% to 84.9% success rates for the hybrid method are not interpretable as accuracy figures.","section":"Section VI, Table II"},{"comment":"The denominator 'Total Acronyms' for the RegEx and hybrid rows is the set of strings flagged by the authors' own regex parser after preprocessing (Listing 1), not a gold-standard set of acronyms in the papers. The paper acknowledges in Section VI that it was 'unable to collect statistics regarding the parser's accuracy' and that the manual review of 40 papers only confirmed that the parser finds all acronyms matching its pattern. The parser can admit false positives (e.g., chemical names, headings) and false negatives (e.g., acronyms starting with lowercase letters or numbers, as noted in Section VII). Consequently, all percentages in Table II are relative to an unvalidated denominator, and the comparison between the hybrid method and GPT-alone rows is not apples-to-apples because GPT-alone uses a different denominator (GPT's own detections).","section":"Section VI, Table II"},{"comment":"The conclusion that 'the combination of our preprocessing steps and parser, alongside GPT-4, yields the best results' outruns the evidence. There are no significance tests, no confidence intervals, and no error analysis linking the reported percentages to actual correctness. Because both the numerator and denominator in Table II are unvalidated, the large gap between the hybrid method (80.5% to 84.9%) and the regex-only method (14.8% to 21.2%) could shrink substantially or disappear if GPT-4's expansions are often wrong. At minimum, the authors need to annotate a stratified random sample of GPT-4 outputs and of parser-identified acronyms against a gold standard, and report precision, recall, and inter-annotator agreement.","section":"Section VII and Section III-C"},{"comment":"The paper reports that preprocessing reduces the number of GPT-identified acronyms (e.g., from 823 to 729 in BC, from 865 to 622 in SB) but does not analyze whether preprocessing removes genuine acronyms along with noise. Since preprocessing is a core component of the proposed method, false negatives introduced by steps such as heading removal, stopword filtering, or the exclusion of acronyms longer than 10 characters should be quantified separately. Otherwise, the reported improvement of GPT+RegEx+Pre over GPT+Pre could be partly due to preprocessing removing hard cases that GPT would have handled correctly.","section":"Section VI, preprocessing discussion"}],"minor_comments":[{"comment":"In the sentence 'presenting an estimate of the average number of acronyms per paper is each domain', the word 'is' should be 'in'.","section":"Section VI-A"},{"comment":"The column header 'Total *' is unclear; the asterisk appears to refer to the note below the table, but the note does not explain the multiplication symbol or the exact meaning of the column.","section":"Table II"},{"comment":"The explanation of code snippet (1) states that 's?' optionally matches an 's' at the end, but the pattern is written as `[A-Z]s?\\b`; the literal 's' is not in a character class, which is correct, but the notation in the explanation could be clearer for readers unfamiliar with regex syntax.","section":"Section IV-B"},{"comment":"The paper alternates between saying 'GPT-4' and 'GPT-4o mini' and states they are used interchangeably; since GPT-4o mini is a distinct model with different behavior, the paper should consistently report the exact model version and date for each experiment.","section":"Section V"},{"comment":"Figure 1 is referenced in Section III-C but is not included in the manuscript text provided; please ensure the figure is present in the final version and that it clearly shows the data flow through preprocessing, regex parsing, and GPT-4 refinement.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is reasonable as an engineering contribution, but the evaluation is the main weakness. I would require a substantially redesigned evaluation with gold-standard annotations before considering publication. The authors should also provide the dataset, code, and raw outputs to make the results verifiable. Given their acknowledgment that manual review was too time-consuming, they may need to sample a subset of papers and annotate a few hundred acronym-expansion pairs rather than claim comprehensive coverage. The lack of comparison with existing acronym-extraction baselines (e.g., Schwartz and Hearst or BERT-based methods) is also a gap, though not the primary reason for my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a straightforward engineering contribution: it combines PDF preprocessing, a regex parser, and GPT-4 to extract acronym-expansion pairs from scientific papers. The pipeline is reasonable, and the authors are honest about many of its limitations. But the central performance claim rests on an evaluation that does not validate the outputs it counts, so the headline numbers should not be taken at face value.\n\nWhat is genuinely useful: the hybrid design is a natural combination not present in the cited literature. The paper gives concrete preprocessing heuristics (removing headers, gene sequences, Roman numerals, etc.) and shows that limiting GPT-4's context to the preceding sentence plus the acronym sentence helps. The evaluation spans 200 arXiv papers across four domains, which is a decent-sized corpus for this kind of work. The authors also explicitly acknowledge that they could not measure the parser's precision and that GPT-4 is nondeterministic. That candor is to their credit.\n\nThe soft spot is the evaluation, and it is load-bearing. Table II reports that GPT+RegEx+Pre finds 72–85% of acronyms while RegEx alone finds about 20%, but the denominator is the set of acronyms identified by the authors' own regex parser, and the numerator counts GPT-4's non-empty outputs as correct without checking them. The prompt explicitly tells GPT-4 to use its best judgment when the full form cannot be determined, so hallucinated expansions are counted as successes. The manual check on 40 papers only verified that the parser finds acronyms matching its own pattern, not that the expansions are correct. There is no gold standard, no precision or error analysis, no significance tests, and no released code or data. The stress-test note is correct: the claimed superiority of the integrated approach over RegEx is not actually established. The method may well be better in practice, but this paper does not show it.\n\nThat said, this is not an incoherent paper. The problem is a fixable evaluation gap, not a fundamentally wrong idea. A serious referee would demand human-annotated gold labels on a sample, precision and recall numbers, error analysis on GPT-4's outputs, and ideally released data/code. With those, the hybrid pipeline could become a useful baseline.\n\nWho is this for? Practitioners building acronym extraction tools or preprocessing pipelines for scientific PDFs. It is not a research breakthrough, but it is a reasonable engineering data point. I would send it to peer review with a clear request for major revisions. I would not cite it in its current form, but I would keep an eye on a revised version.","headline":"A sensible hybrid regex+LLM pipeline whose main claim is undermined by an evaluation that counts unvalidated GPT-4 outputs as correct expansions.","tokens_in":13798,"tokens_out":1444,"would_cite":false,"duration_ms":14584,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A regex parser plus GPT-4 beats either alone at expanding acronyms in scientific PDFs.","keywords":["acronym extraction","acronym expansion","regular expressions","GPT-4","large language models","PDF preprocessing","scientific text mining","natural language processing"],"falsifier":"Take 20 papers from the same four arXiv domains, have a domain expert hand-label every acronym and its correct expansion, then run RegEx, GPT+Pre, and GPT+RegEx+Pre against that gold standard; if most of the expansions GPT-4 supplies are plausible but wrong, or if many acronyms the parser misses are common in the text, the combined pipeline's advantage over a simpler approach largely disappears.","tokens_in":12807,"feed_emoji":"🔤","tokens_out":6353,"duration_ms":50817,"temperature":0.7,"pith_summary":"The paper is trying to establish that acronym-expansion extraction from scientific PDFs is best done by a three-stage pipeline: clean the PDF text aggressively, use a regular-expression parser to identify every acronym-shaped token, and let GPT-4 expand each token using only a two-sentence window of surrounding text. The authors report that across 200 arXiv papers from biochemistry, systems biology, computational linguistics, and numerical analysis, this combined pipeline found expansions for 72.2% to 84.9% of parser-identified acronyms, while the parser alone found only 14.8% to 20.9%. The paper also contends that neither tool alone is adequate: the parser is a reliable detector but a poor expander, and GPT-4 is a fluent expander but an unreliable detector, routinely missing acronyms the parser catches. A sympathetic reader would care because automated acronym expansion improves machine comprehension and readability of scientific text, and the paper proposes a practical division of labor between deterministic pattern matching and a probabilistic language model.","feed_headline":"Hybrid regex-plus-GPT-4 pipeline expands up to 85% of acronyms","feed_subtitle":"Parser finds the acronyms, GPT-4 supplies the full forms: combined accuracy 72-85% vs 15-21% for regex alone.","key_machinery":"The load-bearing mechanism is a two-tier division of labor with limited context. First, a regular-expression parser uses a word-boundary pattern that matches capitalized letter sequences (optionally hyphenated, with an optional trailing 's') to enumerate acronym candidates, and then two hand-built patterns extract expansions written either as 'acronym (expansion)' or 'expansion (acronym)', with stopword filtering and length-based refinement to trim false or overlong expansions. Second, for each acronym the parser could not expand, the algorithm sends GPT-4 the acronym together with at most two sentences — the sentence containing it and the preceding sentence — and instructs the model to refine, infer, or discard the expansion, returning JSON. The design choice that carries the argument is limiting GPT-4's context to two sentences: it keeps prompts small enough to avoid the model's token limits and 'lost-in-the-middle' failures, while giving the model enough local evidence to infer non-local expansions.","core_discovery":"The central claim is that acronym extraction should be split by competence: regular expressions carry recall, GPT-4 carries expansion quality, and preprocessing removes the noise that would otherwise be mistaken for acronyms. On a corpus of 50 papers from each of four arXiv domains, the combined GPT+RegEx+Pre approach expanded 84.9% of parser-identified acronyms in biochemistry, 80.9% in systems biology, 80.5% in computational linguistics, and 72.2% in numerical analysis, compared with 14.8% to 20.9% for the regex parser alone. The paper further reports that GPT-4 on its own returns an expansion for nearly everything it is given, but misses many genuine acronyms that the regex parser finds, such as NTP, Cryo-EM, IDT, and TEV. The authors conclude that as of GPT-4o mini, the most accurate and consistent results come from preprocessing plus parser plus GPT-4.","pith_inferences":["The reported 72-85% figures are computed against the parser's own acronym count, so they are upper-bounded by regex recall; extending the parser to lowercase or digit-leading initialisms (for example 2FA, eLisp) would likely raise the absolute number of acronyms expanded even if the percentage stays similar.","The paper's 'lost-in-the-middle' observation suggests a testable extension: shuffle or randomly order the 15-20 acronyms sent in each GPT-4 prompt and check whether acronyms in the middle of the prompt are expanded less consistently than those at the edges.","A natural validation experiment the paper does not run is a hand-built gold standard of acronym-expansion pairs on a sample of the 200 papers; with one, the same pipeline could be measured for true precision and recall rather than coverage of regex-identified tokens.","The same division of labor — a cheap deterministic detector followed by an LLM resolver over a small context window — likely transfers to other scientific-text extraction tasks such as gene names, chemical entities, or figure references, wherever pattern recall is easy and semantic resolution is hard."],"forward_implications":["A hybrid detector-plus-LLM pipeline can raise acronym expansion coverage from roughly 15-21% with regex alone to 72-85% of parser-identified acronyms, so practical tools for expanding acronyms in PDFs are feasible today.","Because the parser identifies far more acronyms than GPT-4 alone, any LLM-based acronym extractor should run behind a deterministic recall stage rather than asking the model to scan whole documents.","Aggressive PDF preprocessing that strips headers, footers, references, equations, roman numerals, and gene-like strings reduces misidentified acronyms by up to 11%, so text cleanup directly improves downstream expansion accuracy.","GPT-4 can expand acronyms whose definitions are missing or non-local in the text, which is exactly the failure mode that defeats parenthetical-pattern regexes.","Improving preprocessing further should raise the combined pipeline's percentages without changing the model, since GPT-4 currently wastes effort trying to expand misidentified acronyms."],"supporting_citations":[{"why":"Provides the parenthetical abbreviation-detection baseline that the paper's forward/backward regex patterns extend.","marker":"[12]"},{"why":"Shows custom regular expressions can extract abbreviation-meaning pairs, motivating the parser's design.","marker":"[11]"},{"why":"The GPT-4 model whose expansions are evaluated and combined with the parser.","marker":"[9]"},{"why":"Documents the growth of acronyms in scientific literature, framing why automated extraction matters.","marker":"[3]"},{"why":"PyPDF is the tool used to convert PDF manuscripts to plain text for the pipeline.","marker":"[27]"},{"why":"Technical details of GPT-4o mini, including input limits, guide the chunking of acronyms per API call.","marker":"[33]"},{"why":"arXiv is the source of the 200-paper evaluation corpus across four domains.","marker":"[38]"}],"fun_headline_variants":["Regex finds acronyms, GPT-4 expands them: 85% success","Hybrid parser + GPT-4 beats regex alone on acronyms","Preprocessing plus parser plus GPT-4 expands 85% of acronyms","No single tool suffices: hybrid regex-GPT-4 expands 85%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported success rates assume the regex parser's acronym list is the correct denominator and that every GPT-4 output counted in Table II is a correct expansion, but neither assumption was checked against a manually built gold standard.","fun_headline_variants_meta":{"raw":{"variants":["Regex finds acronyms, GPT-4 expands them: 85% success","Hybrid parser + GPT-4 beats regex alone on acronyms","Preprocessing plus parser plus GPT-4 expands 85% of acronyms","No single tool suffices: hybrid regex-GPT-4 expands 85%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1557,"prompt_tokens":1051,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":667,"tokens_out":506,"duration_ms":4762,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:41:07.910377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 20 papers from the same four arXiv domains, have a domain expert hand-label every acronym and its correct expansion, then run RegEx, GPT+Pre, and GPT+RegEx+Pre against that gold standard; if most of the expansions GPT-4 supplies are plausible but wrong, or if many acronyms the parser misses are common in the text, the combined pipeline's advantage over a simpler approach largely disappears.","supporting_citations":[{"cited_title":"A simple algorithm for identifying abbreviation definitions in biomedical text,","cited_arxiv_id":null,"evidence_quote":"Provides the parenthetical abbreviation-detection baseline that the paper's forward/backward regex patterns extend."},{"cited_title":"Automatic extraction of acronym-meaning pairs from medline databases,","cited_arxiv_id":null,"evidence_quote":"Shows custom regular expressions can extract abbreviation-meaning pairs, motivating the parser's design."},{"cited_title":"Available: https://openai.com/research/gpt-4","cited_arxiv_id":null,"evidence_quote":"The GPT-4 model whose expansions are evaluated and combined with the parser."},{"cited_title":"The growth of acronyms in the scientific literature,","cited_arxiv_id":null,"evidence_quote":"Documents the growth of acronyms in scientific literature, framing why automated extraction matters."},{"cited_title":"Available: https://pypdf.readthedocs.io/en/stable/ index.html","cited_arxiv_id":null,"evidence_quote":"PyPDF is the tool used to convert PDF manuscripts to plain text for the pipeline."},{"cited_title":"GPT-4o-mini Technical Report,","cited_arxiv_id":null,"evidence_quote":"Technical details of GPT-4o mini, including input limits, guide the chunking of acronyms per API call."},{"cited_title":"arXiv.org e-Print archive","cited_arxiv_id":null,"evidence_quote":"arXiv is the source of the 200-paper evaluation corpus across four domains."}],"review_version":1}