{"id":"1165a9ca-fbe9-436b-9971-f404195d113f","arxiv_id":"2507.10743","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Custom BERT models pre-trained on adult-service-website ad text outperform generic pre-trained encoders on authorship-verification tasks in that domain.","lead":"This paper trains custom BERT-based language models on a private archive of over 19 million unique texts from adult service websites, and reports that they outperform fine-tuned BERT, RoBERTa, and ModernBERT on an authorship-verification task for such ads. The models are aimed at helping law enforcement and nonprofits analyze ads that may be linked to sex trafficking.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that custom models outperform BERT/RoBERTa/ModernBERT rests on labels derived from the same connected-component graph; the second-order concern that direct identifiers in training text inflate performance is testable and may be the real threat.","rationale":"The reader's weakest_assumption identifies the connected-component label premise as the load-bearing assumption, which is a genuine concern. I agree with it partially, but I think the more actionable and concrete threat is the identifier-memorization channel: the paper states that the proprietary dataset includes post text that naturally contains phone numbers, URLs, and names, and the open dataset distinguishes 'post_masked' from 'post_int', implying that raw post text was not fully de-identified for the training pipeline. The classification dataset is built directly from the same graph, so both training and evaluation share the same label source; that is a circularity concern, but it does not by itself explain why the custom models outperform generic models, since both are trained on the same labels. The identifier-memorization concern, if true, would specifically explain the margin. It is also directly testable via a de-identification ablation, whereas fully external validation of the connected-component labels is much harder and may not be available. I therefore recommend keeping the reader's CONDITIONAL verdict, but with the concrete test focused on de-identification and identifier-masking rather than only external labels. I note the paper does provide real evidence: a detailed architecture description, open dataset for applications, code listing for tokenization, and explicit limitations about data sharing; these do not rescue the evaluation contamination concern but they do support the feasibility of the test. The recommendation is CONDITIONAL rather than REJECT because the core claim could survive de-identification; the test is feasible; and the paper is transparent enough that the result could be repaired.","tokens_in":14669,"tokens_out":2450,"duration_ms":22031,"concrete_test":"Re-run or re-evaluate with a de-identified version of the triplet and classification datasets: replace each phone number with a placeholder token, replace each URL with a special <URL> token, and mask names and site names before pre-training, fine-tuning, and evaluation. If the custom model's advantage over fine-tuned BERT/RoBERTa/ModernBERT shrinks substantially (e.g., by more than half of the reported margin), the headline result is largely an identifier-memorization artifact rather than domain-specific linguistic understanding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparison in Experiment 1 is trained and evaluated on the classification dataset generated by the Section 3.2 graph-connected-component assumption, where each non-giant component is assumed to be a single posting entity. This label-generation process is exactly the structure the models are asked to predict, so part of the reported advantage over generic models could come from learning the graph's shortcut patterns rather than true authorship. The paper attempts to mitigate this by hard negatives and TF-IDF similarity thresholds, but it does not report a sensitivity analysis of the 0.2 hard-negative threshold for the classification dataset or measure how much per-component label noise changes the results. The second, more specific and testable concern is that the unique posts dataset and the triplet/classification data were not scrubbed of direct identifiers: phone numbers, URLs, names, and site names are mentioned as present in the raw ads, and 'post_masked' is only used in the open dataset for applications, not for model training or evaluation. If those identifiers remain in the training and evaluation texts, a model can achieve high accuracy by memorizing identifiers and surface patterns, which would inflate all metrics and disproportionately favor the custom models that see more ASW-specific tokens. This would not falsify a weaker efficiency claim, but it would invalidate the linguistic-understanding interpretation of the headline result and would also explain the large gains from 'dense' custom pre-training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper develops custom BERT-based language models for analyzing text from adult service websites (ASWs). The authors build a graph over post texts and perceptual hashes, treat each non-giant connected component as a single posting entity, and use that assumption to generate a 4M-triplet contrastive dataset and a 1M-pair authorship-verification classification dataset. They pre-train six custom BERT configurations (vocabulary sizes 15,261/30,522/45,783; MLM or WWMLM; sequence lengths 64 or 128) on 19.8M unique post texts, fine-tune sentence transformers on the triplet data, and train binary classifiers on the pair data. They report that the custom models, especially the 20-epoch BERT-New30522-MLM variant, outperform fine-tuned BERT-base, RoBERTa, and ModernBERT on accuracy, recall, F1, and ROC AUC. They also demonstrate three applications: decomposing the giant component of an ASW ad graph, semantic search/clustering, and token/emoji embedding analysis.","tokens_in":14963,"tokens_out":4805,"duration_ms":56484,"significance":"If the reported gains are real, the paper would provide a valuable domain-specific resource for ASW text analysis and a practical recipe for efficient transformer training (short sequences, bfloat16, single-GPU hardware). The scale of the pre-training corpus and the three downstream demonstrations are notable strengths. However, the central comparative claim depends on labels that are generated by the same connected-component construction that the model is later used to predict, and the training/evaluation texts may retain direct identifiers such as phone numbers and URLs. Until these threats are resolved, the results are best interpreted as a proof of concept rather than an established superiority claim. The paper does not ship machine-checked code or an open dataset, and the authors state that data availability will be subject to consultation with law enforcement partners, which limits external reproducibility.","major_comments":[{"comment":"The labels for both the triplet dataset and the classification dataset are defined by the graph-connected-component assumption in Section 3.2: all posts in a non-giant component are assigned to one posting entity, and posts in different components are assigned to different entities. The authorship-verification model is then trained to reproduce exactly that component structure, and Section 7.2 applies it to decompose the same type of graph. If a component actually mixes multiple authors or splits one author across components, the labels contain systematic noise, and the reported accuracy, recall, F1, and ROC AUC measure agreement with the graph construction rather than with true authorship. Please validate the component labels on an external or manually annotated sample (for example, using known phone-number groupings or a small hand-labeled set) and report sensitivity to the hard-negative cosine threshold (0.2) and to the TF-IDF similarity thresholds used when building the graph.","section":"Sections 3.2, 3.3, 5, 6, 7.2"},{"comment":"The unique posts corpus used for pre-training, triplet construction, and classification is not scrubbed of direct identifiers: the paper states that the raw ads contain phone numbers, URLs, names, and site names, and the open application dataset uses a 'post_masked' column specifically to obfuscate numbers. If these identifiers remain in the training and evaluation texts, a model can achieve high performance by memorizing surface patterns rather than by learning linguistic similarity. This would differentially benefit the custom models, which see ASW-specific tokens during pre-training, and would invalidate the interpretation that the improvement reflects linguistic understanding. Please retrain or at least re-evaluate on a version of the corpora with identifiers removed (mirroring post_masked) and report the change in all metrics.","section":"Sections 3.1, 3.2, 7.1"},{"comment":"The headline heatmaps report point estimates without error bars, confidence intervals, or significance tests. The main-effects plots in Figures 4 and 5 show 95% confidence intervals, but the individual model comparisons that support the claim 'custom models outperform RoBERTa and ModernBERT' are not accompanied by any measure of uncertainty. Given that some of the reported differences appear small relative to the spread visible in the aggregate intervals, it is unclear whether the observed advantages are beyond sampling noise. Please provide per-configuration confidence intervals or repeated-run results, and use a multiple-comparison correction if pairwise tests are reported.","section":"Figures 3 and 6, Sections 5 and 6"},{"comment":"The paper does not specify how the 1M-pair classification dataset is split for training and evaluating the binary classifiers, and in Experiment 2 the 60 checkpoint models are evaluated on the same classification dataset that is used to select the recommended configuration. This creates a risk of selection-on-test overfitting: the reported peak performance of BERT-New30522-MLM (20 epochs) may be optimistic because the same data determines both model choice and final metrics. Please describe the train/validation/test split explicitly, and re-evaluate the chosen model on a held-out set that was not used for checkpoint selection.","section":"Sections 5 and 6"}],"minor_comments":[{"comment":"The model name is spelled 'RoBERTA' in the section title and once in the text; it should be 'RoBERTa'.","section":"Section 2.2.2"},{"comment":"The text reads 'Previous research involving AWS data'; this should be 'ASW data'.","section":"Section 4, first paragraph"},{"comment":"The two paragraphs describing the custom model variants repeat the same description of whole word masked language modeling almost verbatim; one of the passages should be removed.","section":"Section 4.2.3"},{"comment":"The caption mentions 'black horizontal reference lines' but the figure is not reproduced in the manuscript text; please ensure the final PDF includes the figure with those reference lines visible.","section":"Figure 2 caption"},{"comment":"The method is called 'LocalMAP' in the text, but the citation title refers to 'locally adjusted graphs'; please clarify whether LocalMAP is the algorithm from reference [23] or a different variant.","section":"Section 7.3"},{"comment":"The expansion 'Adult Services Websites' appears in the abstract, while Section 1 uses 'Adult Services Websites' and later 'ASWs'; please ensure the expansion is consistent throughout.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper handles a sensitive, proprietary dataset and the authors are appropriately cautious about data sharing. My main concern is that the central performance claim is not yet convincingly separated from the label-generation mechanism and from identifier leakage. The suggested sensitivity analyses and held-out evaluation are feasible within the manuscript's scope and would substantially increase confidence. I do not see a need to reject outright; the contributions are potentially useful for the ASW research community, provided the empirical evidence is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is the first real attempt at ASW-specific foundation models, and the engineering story is credible. The headline comparison, though, is weaker than it looks because the labels and the target application come from the same connected-component graph. That does not kill the paper, but it should force a revised evaluation before publication.\n\nWhat's new: roughly 19M unique posts from 240M ads, custom emoji-aware WordPiece tokenizers at three vocabulary sizes, MLM and WWMLM pre-training at 64/128 sequence lengths, new triplet and classification datasets, and a systematic comparison covering TF-IDF, off-the-shelf sentence transformers, fine-tuned BERT/RoBERTa/ModernBERT, and custom models. The efficient pre-training recipe on a single RTX 6000 Ada is a real practical contribution. The three applications—GC decomposition, semantic search, and emoji embedding inspection—are sensible and show intended use, even though they are mostly qualitative.\n\nWhere the argument softens: the classification labels are generated by assuming each non-giant connected component of the post-plus-perceptual-hash graph is one posting entity, and the model is then trained to predict same/different component and later used to decompose that same kind of graph. The reported accuracy, recall, F1, and ROC AUC therefore measure agreement with a structural assumption, not independently validated authorship. The hard-negative threshold of 0.2 and the GC filtering thresholds deserve sensitivity analysis. Second, training texts were not scrubbed of direct identifiers; phone numbers, URLs, and names remain. A model can learn surface matches, which would inflate all metrics and would especially favor custom models that see more ASW-specific tokens. That is testable: mask or remove identifiers and re-run. Until that is done, the linguistic-understanding interpretation is not established. Also, the headline heatmaps in Figures 3 and 6 have no error bars; the paper does report 95% confidence intervals for main effects, but not for the per-configuration comparisons that carry the central claim. Data and models are not released; the authors say they will consider requests, which is understandable given the sensitive domain, but a concrete release plan would strengthen the paper.\n\nThe citation pattern looks fine: relevant ASW and encoder-model work is cited, and the use of the open Freeman dataset for applications is appropriate. This is serious work, not a toy demonstration, and the authors acknowledge several limitations themselves.\n\nWho should read it: people working on trafficking analytics or domain-specific pre-training for noisy, obfuscated, emoji-heavy text. I would send it to peer review, with the clear request that the authors validate on externally labeled authorship data, add identifier-removal ablations, and report uncertainty on the headline numbers.","headline":"First credible ASW-specific pre-trained models, but the headline evaluation is entangled with the graph labels it was built from.","tokens_in":15506,"tokens_out":2378,"would_cite":true,"duration_ms":31322,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Custom BERT models trained on adult-service ad text beat general encoders on authorship verification.","keywords":["adult service websites","sex trafficking","language modeling","authorship verification","BERT","contrastive learning","TF-IDF","emoji analysis"],"falsifier":"Take a held-out set of ads whose true authorship is known from independent records, strip out phone numbers, URLs, and other direct identifiers, and measure whether the custom model still outperforms fine-tuned RoBERTa and ModernBERT; if the gap disappears, the reported gains come from memorizing identifiers rather than from linguistic understanding.","tokens_in":14474,"feed_emoji":"🔍","tokens_out":4464,"duration_ms":49417,"temperature":0.7,"pith_summary":"This paper tries to show that language models pre-trained from scratch on text from adult service websites (ASWs) outperform general-purpose encoders such as BERT-base, RoBERTa, and ModernBERT when all are fine-tuned on the same ASW authorship-verification task. The authors build a graph from 240 million ads using post texts and perceptual image hashes, treat each non-giant connected component as one posting entity, and use those component labels to create 4 million training triplets and 1 million classification pairs. They report that their custom models win on accuracy, recall, F1, and ROC AUC, with the largest gains in recall. The practical payoff is better tools for linking ads, decomposing the giant component in ASW graphs, and understanding coded emoji usage in illicit advertising.","feed_headline":"Custom ASW language models beat BERT, RoBERTa, ModernBERT","feed_subtitle":"Domain-trained encoders lift recall on sex-ad authorship verification, aiding trafficking-related link analysis.","key_machinery":"The machinery is a BERT-base encoder re-built for the ASW domain: a WordPiece tokenizer trained on ASW text at three vocabulary sizes, pre-training with masked language modeling or whole-word masking at sequence lengths of 64 or 128 tokens instead of 512, and a mean-pooling layer fine-tuned with contrastive learning on 4 million anchor-positive-negative triplets. The short sequence length exploits the fact that 99.7% of tokenized posts fit in 64 tokens, cutting self-attention cost enough to train on a single high-end GPU rather than a large cluster. Labels come from a graph whose edges connect posts and perceptual hashes that co-occur in ads; each non-giant connected component is assumed to be one posting entity.","core_discovery":"The central claim is that domain-specific pre-training on 19.8 million unique ASW post texts, using custom WordPiece tokenizers and short sequence lengths, yields sentence-encoder embeddings that are more accurate for ASW authorship verification than fine-tuned general encoders. On a one-million-pair classification dataset, the best custom configuration, BERT-New30522-MLM with a 64-token sequence length and 20 pre-training epochs, reaches performance at or near the top across accuracy, recall, F1, and ROC AUC, and it does so using dense embeddings alone, without sparse ensemble help. The authors also show the learned embeddings can filter erroneous edges in ASW data graphs, retrieve semantically similar ads, and place illicit emojis in a meaningful semantic space.","pith_inferences":["Editorial inference: the same pre-train-then-contrastive-fine-tune recipe may transfer to other short, obfuscated, emoji-heavy text domains, such as drug-market listings or spam, where general-purpose encoders also tend to underperform.","Editorial inference: because direct identifiers like phone numbers and URLs remain in the training text, a portion of the reported improvement could come from memorizing those identifiers rather than from linguistic understanding; a redaction test would separate these signals.","Editorial inference: the early plateau between 3 and 5 pre-training epochs suggests that most of the benefit comes from a small amount of domain adaptation, not from massive additional compute, although the paper's own curves stop at 20 epochs and show some variability.","Editorial inference: the component-as-author label assumption is the main source of label noise, so the method's real-world ceiling depends on how often a connected component actually mixes multiple posting entities."],"forward_implications":["If the claim is correct, ASW-specific models can be trained cheaply and deployed on consumer hardware, with even three pre-training epochs already beating general models on authorship verification.","Practitioners can dismantle the giant component of an ASW data graph into author-coherent clusters by setting a single cosine-similarity threshold on the custom sentence embeddings.","The learned token embeddings can be used to expand known lists of illicit emojis and coded language, since the model groups drug-, dealer-, and commercial-sex-related emojis together in the projected embedding space.","The same contrastive fine-tuning recipe is what delivers the largest measured gains, so domain pre-training plus contrastive fine-tuning is a transferable template for specialized text analysis."],"supporting_citations":[{"why":"Supplies the BERT-base architecture and pre-training tasks that the custom models build on, as well as the bert-base-uncased baseline.","marker":"[16]"},{"why":"Provides the RoBERTa pre-training improvements and the RoBERTa-base model that the custom models are compared against.","marker":"[17]"},{"why":"Introduces ModernBERT, a recent strong encoder-only baseline that the custom models must outperform.","marker":"[18]"},{"why":"Provides the sentence-transformer contrastive learning method used to fine-tune all dense models into similarity-aware encoders.","marker":"[21]"},{"why":"Establishes the perceptual-hash-plus-text graph approach that the paper uses to define posting entities and generate labels.","marker":"[13]"},{"why":"Supplies the open multi-site ASW dataset used in the three application demonstrations: giant-component decomposition, semantic search, and token understanding.","marker":"[14]"}],"fun_headline_variants":["Custom ASW text models outdo BERT, RoBERTa, ModernBERT","Domain-trained transformers edge general encoders on sex-ad text","Efficient custom LLMs beat big encoders on adult ad analysis","Small custom models outperform fine-tuned transformers on ASW text","ASW-specific pre-training wins on authorship verification and more"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training and evaluation pipeline assumes that every non-giant connected component of the post-text and perceptual-hash graph is the work of a single posting entity, so the labels used to train and test the models inherit any mistakes in that grouping.","fun_headline_variants_meta":{"raw":{"variants":["Custom ASW text models outdo BERT, RoBERTa, ModernBERT","Domain-trained transformers edge general encoders on sex-ad text","Efficient custom LLMs beat big encoders on adult ad analysis","Small custom models outperform fine-tuned transformers on ASW text","ASW-specific pre-training wins on authorship verification and more"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1652,"prompt_tokens":1008,"completion_tokens":644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":555}},"tokens_in":624,"tokens_out":644,"duration_ms":7630,"temperature":1.0,"reasoning_tokens":555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:27:08.313711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of ads whose true authorship is known from independent records, strip out phone numbers, URLs, and other direct identifiers, and measure whether the custom model still outperforms fine-tuned RoBERTa and ModernBERT; if the gap disappears, the reported gains come from memorizing identifiers rather than from linguistic understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the perceptual-hash-plus-text graph approach that the paper uses to define posting entities and generate labels."}],"review_version":1}