{"id":"cf3412f8-0563-4cba-8ed6-4d996ab183c2","arxiv_id":"2508.19449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A transformer-based pipeline that fine-tunes a sentence-embedding model on stack traces and ranks duplicate crash reports outperforms prior baselines on four datasets.","lead":"The paper proposes dedupT, a system that adapts a pretrained language model to stack traces and then trains a ranking network to group duplicate crash reports. On four public datasets it reports higher duplicate-ranking accuracy than existing string, retrieval, and deep-learning methods, which could reduce manual triage effort for large software projects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Preprocessing choices (top-N frame count and Java trimming level) appear to be selected on test MRR, which could inflate dedupT's reported margins over baselines.","rationale":"The reader's weakest_assumption focuses on split sensitivity, which is a real concern and is supported by the paper's own Table III. However, I think the more directly load-bearing issue is potential test-set leakage in preprocessing hyperparameter selection: Section VI-D's Figure 8 and Table IX use MRR to guide the choice of N and trimming level, and Table II's dedupT values coincide with the per-dataset best of those choices. Since the paper does not state that these choices were made on a held-out validation split, the main comparison may be inflated in dedupT's favor relative to baselines that were not tuned in the same way. This concern does not require assuming bad faith; it is a missing methodological detail that is easy to check and would settle whether the claimed margins are real. The verdict should remain CONDITIONAL: with the missing split information and no released code or artifacts, the central claim cannot be accepted as-is, but the concern is addressable and the approach is otherwise plausible. I partially agree with the reader because the reader's rationale already mentions the frame-count selection issue, even though the formal weakest_assumption names split sensitivity instead.","tokens_in":20342,"tokens_out":5591,"duration_ms":54357,"concrete_test":"Ask the authors to report which data split was used to generate Figure 8 and Table IX. If the test split was used, rerun dedupT with N and the trimming level fixed a priori (e.g., N=10 and L0) or chosen on the validation split only, for all four datasets and at least five random seeds, then recompute the Table II MRR/RR@k values. Additionally, rerun the Ubuntu comparison with dedupT and Tracesim on the two alternate Table III splits (2250/100/700 and 2450/50/600). If dedupT no longer exceeds PDM/Tracesim by the reported margins, the central ranking claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that dedupT beats ten baselines in MRR/RR@k (Table II). The most load-bearing weakness is that two preprocessing hyperparameters entering that comparison appear to be tuned on the evaluation split itself. Section IV-A says the top-N frame budget is 'determined empirically'; Section VI-D and Figure 8 then report Netbeans/Eclipse MRR for N = 10, 15, 20, and Section VI-D and Table IX report test MRR for trimming levels L0/L1/L2. The final Table II numbers match the per-dataset best of these choices (Netbeans L0: MRR 0.771; Eclipse L2: MRR 0.791), but the paper never states that N and the trimming level were fixed using only a training/validation split before test labels were inspected. If Figure 8 and Table IX were computed on the same test split used in Table II, dedupT's reported MRR benefits from test-set fitting, while the baselines receive no equivalent tuning. That would make the headline margins (e.g., 0.771 vs 0.705 over PDM on Netbeans; 0.791 vs 0.752 over Tracesim on Eclipse) not a clean head-to-head comparison. The split-sensitivity results in Table III add a second, independent concern: Ubuntu MRR ranges from 0.761 to 0.838 across splits, a span larger than the 0.034 margin dedupT holds over Tracesim on the chosen Ubuntu split. Together, these issues mean the reported advantage is not yet established as robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes dedupT, a transformer-based pipeline for stack-trace crash deduplication. It preprocesses stack traces by removing consecutive duplicate frames, retaining a top-N frame budget, cleaning language-specific syntax, and adding positional coding. It then fine-tunes a pretrained sentence embedding model (bge-base) with a multiple-negatives ranking loss on positive/negative stack-trace pairs sampled from training buckets, and aggregates multiple stack traces per report using a parametric max-mean combination. A fully connected network trained with RankNet loss scores candidate reports. The evaluation compares dedupT against ten baselines (PDM/Rebucket, Tracesim, DURFEX, Moroo, PrefixMatching, Brodie, Needleman-Wunsch, TF-IDF, S3M, DeepCrash) on Netbeans, Eclipse, Gnome, and Ubuntu, reporting MRR, RR@k, and ROC-AUC, plus ablation and sensitivity analyses. The central claim is that dedupT improves MRR by over 9% over the best IR baseline and over 15% over the best DL baseline and achieves higher ROC-AUC on unique-crash detection.","tokens_in":20637,"tokens_out":5202,"duration_ms":46539,"significance":"Should the reported gains hold under a clean evaluation protocol, the paper makes a useful contribution: it demonstrates that domain-adapted transformer embeddings improve duplicate-crash ranking over both lexical/alignment methods and prior LSTM-based models on four public datasets. The work is careful in several respects: it uses multiple real-world datasets, includes ten baselines, reports ablations for frame count, trimming level, aggregation method, and embedding model choice, and provides a split-sensitivity analysis. These elements are valuable. The main reservations are methodological: the headline numbers appear to reflect preprocessing hyperparameters chosen on the test split, the main comparison is a single chronological split without confidence intervals or significance tests, and the split-sensitivity analysis shows variance on Ubuntu that is comparable to the reported margin over the best baseline. These issues must be addressed before the claimed advantage is established.","major_comments":[{"comment":"The selection of the frame budget N and the Java trimming level appears to be made on the test split. Section IV-A says N is 'determined empirically' and Figure 8 reports MRR for N=10, 15, and 20; Table IX reports MRR for trimming levels L0/L1/L2. The Table II entries for dedupT match the per-dataset best of these choices (Netbeans L0: MRR 0.771; Eclipse L2: MRR 0.791). The paper does not state that these choices were locked using only the training or validation split before test labels were inspected. If these ablations were computed on the same test split, dedupT's reported margins over baselines (which receive no equivalent per-split tuning) are inflated by test-set fitting. Please rerun the pipeline with N and the trimming level selected on the validation split, or clearly document a separate model-selection split, and report the resulting Table II numbers.","section":"Section VI-D, Tables II and IX, Figure 8"},{"comment":"The headline comparison rests on a single chronological split per dataset. Table III's sensitivity analysis shows that on Ubuntu, dedupT's MRR ranges from 0.761 to 0.838 (SD 0.032) across three splits, a spread larger than the 0.034 MRR margin over Tracesim on the chosen split (0.786 vs 0.752). Without confidence intervals or significance tests across multiple splits, the reported ranking advantages may not be robust. Please report results across the three splits for dedupT and at least the top baselines, or provide bootstrap confidence intervals for the Table II comparisons.","section":"Section V-B and Table III"},{"comment":"Negative pairs for embedding-model adaptation are constructed by sampling from 'a random subset of stack traces drawn from the top 50 buckets identified by Lerch and Mezini [22]'. It is unclear whether this bucket list was derived from the full datasets including the test period. If so, the fine-tuning of the embedding model (and the classifier negative sampling) could leak test information. Please clarify the provenance of this list and, if needed, restrict it to training-period data.","section":"Section IV-B"},{"comment":"DeepCrash was reimplemented because no public code was available and then modified for ranking; small implementation differences can change relative performance. Please provide the reimplementation and a validation that it reproduces the published behavior on the original clustering task, or make the code available. This is needed to trust the DL comparisons in Table II.","section":"Section V-A"}],"minor_comments":[{"comment":"There is a typo in the stack-frame-selection paragraph: 'Exceeding this may limit may result in incomplete frame' should be 'Exceeding this limit may result in an incomplete frame'. Also, the RQ3 heading contains 'perfoms' instead of 'performs'.","section":"Section VI-D"},{"comment":"The symbol '∥' is used for concatenation in Equation 4 but is not defined; please define it explicitly as the concatenation operator.","section":"Equation 4"},{"comment":"The embedding-quality evaluation reports Pearson, Spearman, and Euclidean similarities, but the text does not state which split (training, validation, or test) these correlation values are computed on; please clarify.","section":"Section V-C"},{"comment":"The comparison with LLM-based pipelines uses the full test set for Ubuntu but only 200 samples for NetBeans; this makes the cross-dataset comparison difficult to interpret. Please state the sample-size limitation explicitly and consider reporting standard errors for the smaller sample.","section":"Table X"},{"comment":"The main comparison table does not report standard deviations or significance tests for any method. Since the sensitivity analysis shows non-negligible variance on Ubuntu, adding at least bootstrap confidence intervals for the key rows would substantially strengthen the claims.","section":"Section VI-A, Table II"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful empirical contribution, but the current evaluation protocol is not yet convincing. If the authors re-run with validation-based hyperparameter selection, multi-split confidence intervals, and a clarified negative-sampling procedure, I would be willing to see a revision. I also note the absence of a code/artifact link, which is increasingly expected for empirical software-engineering papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick read on arXiv:2508.19449 (dedupT).\n\nThe genuinely new piece is using a contrastively fine-tuned sentence transformer to embed whole stack traces, plus a parametric max-mean pooling to combine multiple traces per crash report. That's a sensible adaptation, and it pays off: on four public datasets they beat ten baselines including Tracesim, PDM, S3M, and DeepCrash, often by 15%+ in MRR over the best DL baseline. The extensive ablation (embedding model choice, aggregation, frame truncation, LLM comparison) is a real strength. The paper is also honest about reimplementing DeepCrash and about not releasing code.\n\nThe soft spot is real and load-bearing for the headline numbers. The preprocessing choices—the number of top frames N and the Java trimming level (L0/L1/L2)—appear to be selected on the test split itself. Section VI-D reports test MRR for N=10/15/20 and for the three trimming levels, and the Table II numbers match the per-dataset best (Netbeans L0, Eclipse L2). The paper never states that N and trimming were fixed using only a training/validation split. So dedupT effectively gets a per-dataset oracle for these two choices, while the baselines don't. That can easily account for a few MRR points, which is the whole margin here.\n\nSecond, the split sensitivity analysis (Table III) is honest but worrying: on Ubuntu, MRR ranges 0.761–0.838 across three chronological splits. The chosen Ubuntu split gives 0.786, only 0.034 above Tracesim's 0.752. So the advantage on Ubuntu is fragile, and the single-split headline doesn't give confidence intervals or significance tests.\n\nAlso missing: any released artifacts, and the contrastive fine-tuning hyperparameters (epochs, learning rate, temperature) are not reported. Reproduction would be painful.\n\nNone of this kills the central idea. The fine-tuned embeddings clearly help (Table V and VI), and even a conservative reading leaves a useful result. But the reported margins are optimistic, and the test-set tuning makes the comparison to baselines unfair as-is.\n\nWho should read it: anyone working on crash deduplication or applying sentence transformers to structured or semi-structured logs. It deserves a serious referee—send it to review with a major-revision request to redo the hyperparameter selection on a validation split, report split-wise results with intervals, and release code/data.\n\nRecommendation: engage with it; it's a worthwhile empirical contribution once the tuning protocol is cleaned up.","headline":"The core idea is sensible and the empirical study is broad, but preprocessing hyperparameters appear to be selected on the test split, making the reported margins optimistic.","tokens_in":21169,"tokens_out":3946,"would_cite":false,"duration_ms":34624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stack-trace-tuned transformer ranks duplicate crash reports up to 9% better than the best alignment method.","keywords":["crash deduplication","stack trace","transformer adaptation","contrastive learning","Mean Reciprocal Rank","crash report triage","pretrained language model","duplicate crash reports"],"falsifier":"A reader could rerun dedupT and the strongest baselines (Tracesim and PDM) on several chronological splits of the four datasets, especially Ubuntu, and compare MRR with confidence intervals. If on a split outside the reported ones dedupT no longer beats Tracesim—the reported Ubuntu margin of 0.034 is smaller than the paper's own split-to-split spread of 0.077—the ranking advantage would not be confirmed.","tokens_in":20140,"feed_emoji":"🧩","tokens_out":9095,"duration_ms":76542,"temperature":0.7,"pith_summary":"Automated crash reporting systems generate many duplicate reports, and deduplicating them from stack traces is a ranking problem: for each new report, list the already-known reports most likely to be the same crash. The paper proposes dedupT, a transformer-based pipeline that treats a stack trace as one contextual passage rather than a set of isolated frames. It fine-tunes a pretrained sentence-embedding model on stack traces with a contrastive loss, then trains a small fully-connected network to rank duplicate reports. On four public datasets (Netbeans, Eclipse, Gnome, Ubuntu) dedupT reports higher Mean Reciprocal Rank than all ten baselines—often over 15% higher than the best deep-learning baseline and up to 9% higher than traditional methods—and higher ROC-AUC for spotting reports with no known duplicate. If that holds, crash triage could become more accurate while relying only on stack traces.","feed_headline":"Stack-trace transformer lifts duplicate-crash ranking by up to 9%","feed_subtitle":"Whole-trace embeddings, tuned on stack data, beat string alignment and LSTM baselines on four real crash datasets.","key_machinery":"The load-bearing mechanism is the adapted embedding model: a pretrained sentence-transformer embedding model (a model that maps text to a fixed vector; the default is bge-base) fine-tuned on stack-trace pairs with Multiple Negatives Ranking Loss, a contrastive objective that pulls duplicate traces together and pushes non-duplicates apart. Around this core, dedupT stacks three further components: a preprocessor that removes duplicate frames, keeps the top ten frames, and adds positional coding; a parametric max-mean aggregation that combines the most similar stack-trace pair's embeddings with the mean embedding of all traces in a report, weighted by a learnable parameter; and a two-layer fully-connected classifier, trained with RankNet loss, that consumes the difference, mean, and element-wise product of the two report embeddings and outputs a duplication score.","core_discovery":"dedupT's central claim is that stack traces should be embedded as whole, ordered passages, with positional coding and contrastive fine-tuning, rather than analyzed frame-by-frame or subframe-by-subframe as in LSTM-based predecessors. The paper argues that a pretrained language model fine-tuned on stack-trace pairs learns an embedding space in which duplicate traces sit closer together even when exact frame alignment fails, and that combining the best-matching stack trace with the mean of all traces in a report gives a better report-level representation than either alone. The result is a duplication score from a two-layer fully-connected network trained with RankNet loss. Across the four datasets, dedupT reports the highest MRR and RR@k of all methods compared, and ROC-AUC above alignment and information-retrieval baselines; DeepCrash has a higher ROC-AUC on Gnome, but a much lower RR@1, so the paper argues the practical ranking advantage remains with dedupT.","pith_inferences":["Beyond the four datasets, the same adaptation recipe should transfer to other statically typed languages and to reports that mix stack traces with logs, because only the frame-cleaning stage is language-specific; the paper does not test this.","The reported latency gap between dedupT (~2s per query) and the LLM reranking pipeline (~30s) suggests the most practical division of labor is dedupT for first-stage retrieval followed by an LLM on only the top candidates; the paper lists this as future work, not a demonstrated result.","The Ubuntu result—the largest adaptation gain on the dataset with the most specialized naming conventions—implies the method's advantage may grow as traces move further from natural-language vocabulary; that is a testable prediction, not a claim the paper makes.","ROC-AUC and RR@1 can disagree, as the Gnome DeepCrash numbers show; a fair deployment comparison should fix the target operating point (top-1 vs top-10 ranking) before choosing a model."],"forward_implications":["Stack-trace-only deduplication can be built from an open, fine-tunable embedding model plus a small classifier, avoiding the hours of per-project hyperparameter tuning that alignment methods require; the paper notes TraceSim tuning alone took over 12 hours on Netbeans.","Which pretrained model is chosen matters less than the stack-trace fine-tuning itself, so teams can swap in whatever sentence-embedding model fits their deployment constraints.","Multi-stack reports are handled without concatenating traces: parametric max-mean pooling outperforms max pooling, mean pooling, and multi-head attention in the paper's ablation on Netbeans and Eclipse.","The same pipeline also flags unique crashes, with ROC-AUC above all alignment and information-retrieval baselines and competitive with LSTM models, though the paper notes DeepCrash has higher ROC-AUC on Gnome but far lower RR@1.","Embeddings can be precomputed and cached, so inference stays cheap enough for live triage even before approximate-nearest-neighbor indexing is added."],"supporting_citations":[{"why":"Supplies the TraceSim baseline, the shared evaluation pipeline, dataset statistics, and the multi-stack-trace scenario that dedupT is designed for.","marker":"[2]"},{"why":"Supplies the S3M LSTM baseline, the negative-pair sampling strategy, and the RankNet classifier training setup dedupT adapts.","marker":"[6]"},{"why":"Supplies the pretrained sentence-embedding model family whose embeddings dedupT fine-tunes and then feeds to the classifier.","marker":"[19]"},{"why":"Supplies the duplicate-frame removal heuristic used in dedupT's stack-trace preprocessing.","marker":"[4]"},{"why":"Supplies the stack-trace extraction and the top-bucket sampling list used to form negative training pairs.","marker":"[22]"},{"why":"Supports the 'top frames matter most' premise that justifies keeping ten frames rather than full traces.","marker":"[7]"},{"why":"Supplies the Multiple Negatives Ranking Loss used for contrastive fine-tuning of the embedding model.","marker":"[31]"},{"why":"Supplies the RankNet loss used to train the duplicate-ranking classifier.","marker":"[34]"},{"why":"Supplies the contrastive-learning principle that the embedding adaptation step operationalizes.","marker":"[28]"}],"fun_headline_variants":["Transformer adapts to stack traces for sharper crash dedup","Whole-trace embeddings beat frame-by-frame crash dedup models","Adapted PLMs deduplicate crashes 15% better MRR","Stack-trace BERT: holistic embeddings cut duplicate crash triage","DedupT: transformer ranking outperforms LSTM and string methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparisons assume the single chronological train/validation/test split chosen per dataset is representative of how crash reports arrive; the paper's own sensitivity table shows Ubuntu MRR varying from 0.761 to 0.838 across splits, a spread larger than dedupT's reported 0.786-versus-0.752 margin over the closest baseline on that dataset.","fun_headline_variants_meta":{"raw":{"variants":["Transformer adapts to stack traces for sharper crash dedup","Whole-trace embeddings beat frame-by-frame crash dedup models","Adapted PLMs deduplicate crashes 15% better MRR","Stack-trace BERT: holistic embeddings cut duplicate crash triage","DedupT: transformer ranking outperforms LSTM and string methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1339,"prompt_tokens":971,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":278}},"tokens_in":587,"tokens_out":368,"duration_ms":3600,"temperature":1.0,"reasoning_tokens":278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:53:22.147318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun dedupT and the strongest baselines (Tracesim and PDM) on several chronological splits of the four datasets, especially Ubuntu, and compare MRR with confidence intervals. If on a split outside the reported ones dedupT no longer beats Tracesim—the reported Ubuntu margin of 0.034 is smaller than the paper's own split-to-split spread of 0.077—the ranking advantage would not be confirmed.","supporting_citations":[{"cited_title":"Tracesim: An alignment method for computing stack trace similarity,","cited_arxiv_id":null,"evidence_quote":"Supplies the TraceSim baseline, the shared evaluation pipeline, dataset statistics, and the multi-stack-trace scenario that dedupT is designed for."},{"cited_title":"S3m: Siamese stack (trace) similarity mea- sure,","cited_arxiv_id":null,"evidence_quote":"Supplies the S3M LSTM baseline, the negative-pair sampling strategy, and the RankNet classifier training setup dedupT adapts."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained sentence-embedding model family whose embeddings dedupT fine-tunes and then feeds to the classifier."},{"cited_title":"Quickly finding known software problems via automated symptom matching,","cited_arxiv_id":null,"evidence_quote":"Supplies the duplicate-frame removal heuristic used in dedupT's stack-trace preprocessing."},{"cited_title":"Do stack traces help developers fix bugs?","cited_arxiv_id":null,"evidence_quote":"Supports the 'top frames matter most' premise that justifies keeping ten frames rather than full traces."},{"cited_title":"Efficient natural language response suggestion for smart reply,","cited_arxiv_id":null,"evidence_quote":"Supplies the Multiple Negatives Ranking Loss used for contrastive fine-tuning of the embedding model."},{"cited_title":"Learning to rank using gradient descent,","cited_arxiv_id":null,"evidence_quote":"Supplies the RankNet loss used to train the duplicate-ranking classifier."}],"review_version":1}