{"id":"b0cb1376-2b7b-47af-9c6d-1191e45ab120","arxiv_id":"2506.03793","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Cadence is a multilingual punctuation restoration model for 22 Indian languages and English that adapts Gemma-3-1B with bidirectional attention and Masked Next Token Prediction, reporting state-of-the-art macro F1 against two older baselines.","lead":"A team from AI4Bharat introduces Cadence, a punctuation restoration model for 22 Indian languages and English, built by turning the Gemma-3-1B language model into a bidirectional tagger and fine-tuning it on a new multilingual corpus. The model reports large F1 gains over older IndicPunct and DeepMultilingualPunctuation baselines, especially on formal written text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on a Gemini-filtered test set and two old baselines; without unfiltered/human-annotated comparison the reported margins may be inflated.","rationale":"The reader's weakest assumption identifies the most load-bearing issue: the SOTA claim is a numeric claim, and every number in Table 3 comes from a test set that was post-hoc filtered by an LLM judge. The paper openly describes this in Section 4.3 but never quantifies the filtering or shows that relative rankings are stable on unfiltered data. Because Cadence was trained on the same corpora and on LLM-generated punctuation, the filter can selectively keep examples that match its training distribution, inflating the margin over older baselines. I do not see an internal inconsistency in the architecture or training: the MNTP bidirectional adaptation, curriculum, and 30-class label space are coherent, and the model is a plausible contribution. The evaluation protocol is the weakest link. A human-annotated unfiltered benchmark would settle it. Two adjacent concerns reinforce the same conclusion rather than replacing it: no reported contamination check between FineWeb-2 and the test corpora, and only two older baselines without error bars or significance tests. These considerations make the current evidence suggestive, not definitive, so the conditional verdict is appropriate.","tokens_in":16696,"tokens_out":10942,"duration_ms":109238,"concrete_test":"Take the original, pre-filter held-out instances from IndicCorp-v2, Sangraha-Verified, translated Cosmopedia, and IndicVoices (e.g., 300 per language/domain, including all Gemini-low-scoring sentences) and have native-speaker annotators provide gold punctuation. Compute Cadence, IndicPunct, and DeepMultilingualPunctuation macro-F1 on this unfiltered human-annotated set and compare with Table 3. Also report the fraction of instances removed by the Gemini filter and the per-label F1 breakdown. If Cadence's margin over baselines persists on the unfiltered set, the filtering concern is not decisive; if the margin shrinks or disappears, the SOTA claim is an artifact of test-set construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence for the SOTA claim is Table 3, but it is produced under an evaluation protocol that can inflate Cadence's advantage. Section 4.3 states that held-out test instances were filtered by Gemini-2.5-Flash using a punctuation-quality rubric, retaining only instances scoring 4.5 or higher. The paper reports no counts of how many instances were removed per language or domain, no unfiltered results, and no human validation. Because Cadence was fine-tuned on the same four corpora described in Section 3.1, and much of that data was punctuated by Llama-3-405B, filtering to 'high-quality' examples may select for the exact distribution Cadence was trained on; the LLM judge may also share stylistic preferences with that training data. This does not reveal an architectural inconsistency, but it threatens the external validity of the central SOTA claim. An adjacent unresolved issue is that no contamination check between the FineWeb-2 pretraining data and the test corpora is reported, so memorization cannot be excluded. The most direct way to settle this is to re-evaluate on unfiltered, human-annotated held-out instances.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Cadence, a punctuation-restoration model for English and 22 Indian languages. It starts from Gemma-3-1B-Pretrain, converts the attention to bidirectional, continues pretraining on FineWeb-2 Indic data with a Masked Next Token Prediction objective under a four-phase curriculum, and fine-tunes on a mixture of Sangraha-Verified, IndicVoices-ST, translated Cosmopedia, and IndicCorp-v2 to classify 30 punctuation labels per token. Evaluation uses macro-F1 on held-out splits of the four fine-tuning corpora plus BPCC, after filtering test instances with Gemini-2.5-Flash to keep only those scoring at least 4.5 on a punctuation-quality rubric. The authors report substantial gains over IndicPunct and DeepMultilingualPunctuation on a reduced set of 'focus labels', plus zero-shot results on Bhojpuri and low-resource results on Manipuri, and they release the model.","tokens_in":16784,"tokens_out":5284,"duration_ms":45958,"significance":"The main contribution is a single open-source model covering 23 languages with 30 punctuation classes, including Indic-specific marks such as the danda and Urdu full stop, which is more useful than the previous 14-language IndicPunct. The architecture choice, adapting a pretrained decoder into a bidirectional MNTP tagger, is a reasonable and reproducible design, and the release of Cadence is a practical asset for Indic NLP pipelines. The analysis of performance versus training-data size and the identification of persistent weaknesses on spontaneous speech are informative. However, the central SOTA claim is only as strong as the evaluation, and the evaluation has three load-bearing weaknesses: an LLM-filtered test set without unfiltered results or human validation, only two pre-LLM baselines, and no statistical significance or error-bar analysis. With those issues addressed, the empirical claim would be credible; as it stands, the SOTA claim is not fully supported.","major_comments":[{"comment":"The held-out test set is filtered by Gemini-2.5-Flash using a 1-5 punctuation-quality rubric, and only instances scoring 4.5 or higher are retained. The paper does not report how many instances were removed per language or domain, does not report scores on the unfiltered set, and provides no human-agreement analysis for the judge. Because the training data include Llama-3-405B-generated punctuation (Section 3.1, IndicVoices-ST and translated Cosmopedia), an LLM judge may share stylistic preferences with the model's training distribution, so filtering can select examples on which Cadence is unusually strong. Please report filter statistics, re-run the comparison on the unfiltered held-out set, and provide a human-validated subset or a detailed disagreement analysis.","section":"Section 4.3"},{"comment":"The SOTA comparison is against IndicPunct (2022) and DeepMultilingualPunctuation (2021), both with small label spaces and no LLM-based component, and the paper presents no error bars, confidence intervals, or significance tests. The claim of new state-of-the-art is therefore based on a single run against two old baselines, and the reported margins (e.g., 0.78 vs 0.26 on Cosmopedia) may reflect evaluation-protocol differences rather than model quality. Add at least one recent multilingual or LLM-based baseline, and report bootstrap confidence intervals or multi-seed variance for Cadence and the baselines.","section":"Section 5.1 and Table 3"},{"comment":"The aggregate scores are not computed over the same language sets. Cadence supports all 23 languages, whereas IndicPunct and DeepMultilingualPunctuation support only subsets; cells marked 'x' or '–' are excluded differently for each model. The headline '0.76 vs 0.54' on IndicCorp-v2 therefore does not compare like with like. Report averages over the common subset of languages or use per-language paired differences with significance tests.","section":"Table 3, Overall row"},{"comment":"Test instances are held out from the same four corpora used for fine-tuning, but the paper does not describe the split procedure, any deduplication, or whether documents or paragraphs overlapping between train and test were removed. In addition, continual pretraining uses FineWeb-2 (Section 4.1.1), and no contamination check against the test corpora is reported, so memorization cannot be ruled out. Please specify the splitting and deduplication protocol and, if feasible, report a contamination analysis.","section":"Sections 3.1 and 4.1"}],"minor_comments":[{"comment":"The column headers ('Formal Extempore S IC C BPCC IV S IC C IV IC BPCC') are difficult to parse; provide a legend or separate sub-tables so it is clear which columns correspond to each model and corpus.","section":"Table 3"},{"comment":"There is an unresolved cross-reference 'detailed in Table ??' that should read Table 2.","section":"Section 4.2.3"},{"comment":"The model name is styled inconsistently as 'GEMMA 3-1B-PRETRAIN', 'GEMMA -3-1B', and 'Gemma-3-1B-Pretrain'; use the official capitalization consistently.","section":"Throughout"},{"comment":"The text reports 'Sanskrit (75k, 0.2966)' and 'Sindhi (58k, 0.2966)', but Table 4 lists written all-labels scores of 0.21 and 0.29 for these languages; align the prose and table values.","section":"Section 5.3"},{"comment":"The Limitations section discusses training-data quality but does not mention that the evaluation test set itself was LLM-filtered; this is a separate limitation that should be acknowledged.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the difficulty of obtaining human-validated test data for 23 languages, and the resource contribution is real. My main reservation is that the central SOTA claim is currently supported only by an evaluation protocol that can systematically favor Cadence. If the authors provide unfiltered results, filter statistics, and at least one stronger recent baseline, I would be willing to support acceptance. I would also ask the editor to ensure the train/test overlap and FineWeb-2 contamination issues are addressed, since the pretraining corpus is drawn from the same broad web space as some evaluation sources."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Cadence is a real contribution—a 30-label, 23-language punctuation restoration model built by adapting Gemma-3-1B with bidirectional attention and MNTP, plus a new aggregated training corpus. The modeling and data work are solid and useful. The evaluation, however, is the weak link. The headline numbers in Table 3 come from a test set filtered by Gemini-2.5-Flash to keep only instances scoring 4.5/5 on punctuation quality, and there are no unfiltered results, no human validation, no error bars, and no contamination check against FineWeb-2. That doesn't kill the paper, but it means 'state of the art' is not established. The margins over IndicPunct and DMP might partly reflect the judge's taste aligning with the LLM-generated training data.\n\nWhat's genuinely new: the corpus aggregation across Sangraha, IndicVoices, Cosmopedia, and IndicCorp-v2; the 22-language coverage; and the 30-label space including danda, Urdu marks, and Santali mucaad. The curriculum pretraining with MNTP and the weighted sampling are sensible choices. The per-language analysis and zero-shot results on Bhojpuri and Manipuri are interesting. The paper is honest about limitations: it admits spontaneous speech lags, rare marks are hard, and data volume alone doesn't explain performance.\n\nThe main soft spot is the test-set filtering. Section 4.3 is transparent, but it introduces selection bias. If Gemini's notion of 'good punctuation' aligns with the Llama-3-405B punctuation used to create IndicVoices and Cosmopedia, filtering to high scores selects for the training distribution. The paper doesn't report how many instances were removed per language or domain, nor does it compare on the unfiltered set. A human-annotated subset, even a few hundred sentences per language, would settle this. Also, only two baselines, neither recent, and no significance tests. That said, the architecture is sound and the model likely does help in practice; the magnitude of the SOTA claim is just not proven.\n\nWho this is for: NLP researchers working on Indic MT/TTS/ASR pipelines. They'll find Cadence useful regardless of the exact F1. It deserves a serious referee, but the evaluation needs to be strengthened before the SOTA claim is accepted.","headline":"A genuinely useful model and corpus for Indic punctuation, but the SOTA claim is not yet supported because the test set was filtered by an LLM judge and compared only against two old baselines.","tokens_in":17463,"tokens_out":3029,"would_cite":true,"duration_ms":27310,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single model, Cadence, outperforms prior punctuation-restoration systems across 22 Indian languages and English.","keywords":["punctuation restoration","multilingual NLP","Indic languages","speech transcripts","sequence tagging","masked next token prediction","low-resource languages","LLM adaptation"],"falsifier":"Take the full held-out set before the judge filters out low-scoring sentences, have native speakers of a sample of the 22 languages score a random subset, then compute Cadence's macro-F1 and the baselines' macro-F1 on that unfiltered subset; if the performance gap over the baselines shrinks or disappears, the SOTA claim depends on the automated judge's filtering.","tokens_in":16340,"feed_emoji":"📝","tokens_out":6921,"duration_ms":63556,"temperature":0.7,"pith_summary":"The paper claims that one relatively small pretrained language model, adapted into a bidirectional token classifier, can restore punctuation in all 22 scheduled Indian languages plus English, and that it beats the previous state of the art on both clean written text and spontaneous speech transcripts. If true, this collapses a patchwork of language-specific punctuation systems into a single checkpoint, which matters because unpunctuated ASR output and web-crawled text are a bottleneck for downstream translation, summarization, and speech synthesis. The authors build Cadence by taking a 1-billion-parameter causal language model, switching its attention to bidirectional, pre-training it with a masked next-token-prediction objective on Indic web data, and fine-tuning it to label each token with one of 30 punctuation classes.","feed_headline":"One model now punctuates 22 Indian languages and English","feed_subtitle":"Cadence beats earlier tools on both formal text and messy speech transcripts, including low-resource Indic languages.","key_machinery":"Masked Next Token Prediction (MNTP): after converting the causal attention of Gemma-3-1B to bidirectional, the model is pre-trained to predict a masked token from the representation of the immediately preceding unmasked token. This objective trains the model to use local left-context to decide what comes next, which is exactly the skill punctuation restoration needs, since a punctuation mark is predicted from the token it follows. The second mechanism is a four-phase curriculum over languages, moving from English to high-resource, then low-resource Indic languages, then all 23 languages together, which lets the model transfer knowledge to scarce-data languages. Fine-tuning then replaces the language-modeling head with a linear classifier over 30 punctuation labels, trained with weighted sampling to upweight low-resource languages.","core_discovery":"Cadence establishes a new state of the art for punctuation restoration in Indic languages across both formal text and extempore speech transcripts. On the 14 languages where the earlier IndicPunct system applies, Cadence scores higher on every shared test set, and it extends coverage to the remaining eight scheduled languages and English. The central result is that a generalist, non-autoregressive model trained with a curriculum over languages, then fine-tuned on a mixture of verified web text and LLM-punctuated speech transcripts, transfers well to low-resource languages and even to a completely unseen language. The model's overall macro-F1 on focus punctuation labels is 0.79 on written text and 0.62 on spontaneous transcripts, with the gap reflecting the inherent difficulty of disfluent speech.","pith_inferences":["The same bidirectional MNTP recipe could be applied to other token-level tasks for low-resource languages—named-entity recognition, part-of-speech tagging, or inverse text normalization—reusing Cadence's pre-training rather than starting from scratch.","The 0.17 F1 gap between written text and spontaneous transcripts suggests the bottleneck is disfluency structure, not punctuation knowledge; a model explicitly trained on disfluency-tagged utterances might close much of that gap.","Because the test set was filtered by an automated punctuation judge, an independent human-annotated gold set on a random sample would test whether the SOTA claim survives a judge-agnostic evaluation.","The strong Santali result despite low data hints that script-specific punctuation conventions can be learned from very few examples when the model already shares structure with related languages; extending the label set to other tribal languages' marks is a natural next step."],"forward_implications":["Downstream Indic NLP pipelines—machine translation, text-to-speech, and ASR post-processing—can adopt a single Cadence checkpoint instead of maintaining language-specific punctuation models.","The model covers Indic-specific marks such as the Devanagari danda, the Urdu full stop, Arabic commas, and the Santali mucaad, so those writing systems get punctuated in their own conventions, not translated English punctuation.","Reported gains over baselines hold on spontaneous speech transcripts, meaning ASR output with false starts and repetitions is already better served than by prior tools.","Low-resource languages like Bodo, Dogri, and Santali receive usable punctuation restoration despite tiny training corpora, thanks to cross-lingual transfer during curriculum pre-training.","The zero-shot Bhojpuri result suggests the same model can be dropped into additional unseen Indic languages without retraining, though the absolute score is modest."],"supporting_citations":[{"why":"Supplies the Masked Next Token Prediction objective used to adapt the causal language model into a bidirectional encoder.","marker":"BehnamGhader et al., 2024"},{"why":"The Gemma-3-1B pretrained model that Cadence modifies and fine-tunes.","marker":"Team et al., 2025"},{"why":"Sangraha-Verified corpus contributes web-scraped, verified punctuated text to the fine-tuning mixture.","marker":"Khan et al., 2024"},{"why":"IndicVoices provides spontaneous speech transcripts that are LLM-punctuated and used as training data for spoken-style text.","marker":"Javed et al., 2024"},{"why":"Translated Cosmopedia adds large-scale general-knowledge text across Indic languages to the fine-tuning set.","marker":"Ben Allal et al., 2024"},{"why":"IndicCorp-v2 supplies a large Indic corpus used both for training and for held-out test instances.","marker":"Doddapaneni et al., 2023"},{"why":"FineWeb-2's Indic subset is the continual pre-training corpus and the source of zero-shot Bhojpuri test text.","marker":"Penedo et al., 2024"},{"why":"IndicPunct is the main multilingual baseline for 14 Indian languages that Cadence claims to surpass.","marker":"Gupta et al., 2022"},{"why":"DeepMultilingualPunctuation is the baseline used for English and for the focus-label comparison.","marker":"Guhr et al., 2021"}],"fun_headline_variants":["Punctuation AI now covers 22 Indian languages and English","Cadence sets new SOTA for Indic punctuation, including speech","Zero-shot punctuation for 8 more Indic languages plus English","From 14 to 22: Cadence masters Indic punctuation and disfluent speech","One punctuation model beats prior SOTA across 22 Indic languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark is built by keeping only held-out test sentences that an AI judge scores at 4.5 or higher on punctuation quality, and if that judge's preferences line up with how large language models punctuate, the reported F1 scores—and the state-of-the-art claim—could be higher than they would be against a strictly human-graded test set.","fun_headline_variants_meta":{"raw":{"variants":["Punctuation AI now covers 22 Indian languages and English","Cadence sets new SOTA for Indic punctuation, including speech","Zero-shot punctuation for 8 more Indic languages plus English","From 14 to 22: Cadence masters Indic punctuation and disfluent speech","One punctuation model beats prior SOTA across 22 Indic languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1220,"prompt_tokens":868,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":484,"tokens_out":352,"duration_ms":3409,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:54:40.250343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the full held-out set before the judge filters out low-scoring sentences, have native speakers of a sample of the 22 languages score a random subset, then compute Cadence's macro-F1 and the baselines' macro-F1 on that unfiltered subset; if the performance gap over the baselines shrinks or disappears, the SOTA claim depends on the automated judge's filtering.","supporting_citations":[],"review_version":1}