Pith. sign in

REVIEW 5 major objections 9 minor 41 references

SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation

T0 review · 5 major / 9 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SMAB claims word-level sensitivity can be estimated cheaply and without labels, and shows it predicts accuracy drops and sharpens attacks.

desk verdict The sensitivity estimator is a running average of an OR of two flip events, not Hahn-style sensitivity; the paper's applications are interesting but rest on that mismatch. read the letter →

arxiv 2502.07101 v1 pith:SCYTOS6R submitted 2025-02-10 cs.CL

Saurabh Kumar Pandey , Sachin Vashistha , Debrup Das , Somak Aditya , Monojit Choudhury This is my paper · ORCID
classification cs.CL
keywords wordsensitivitymulti-armedbandittextclassificationadversarialgenerationunsupervisedaccuracyproxymaskedlanguagemodelingmodelinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SMAB (Sensitivity-based Multi-Armed Bandit) claims that word-level sensitivity of a text classifier—how likely replacing a word is to change the predicted label—can be estimated cheaply and without gold labels, using a two-level bandit that samples sentences, replaces target words with masked-language-model alternatives, and counts label flips. The paper argues this estimate is good enough to separate words that matter from words that do not in template-generated tests, to serve as an unsupervised proxy for accuracy when gold labels are missing, and to sharpen adversarial text attacks. In the paper's experiments, sensitivity-guided prompts raise attack success rate by 15.58 percentage points over the perturbation-prompt baseline (the introduction states 13.61), and adding a sensitivity reward improves paraphrase attack quality by 12.00 percentage points over the best baseline. If true, sensitivity—previously requiring exponential enumeration—becomes a practical black-box diagnostic for sequence classifiers.

What carries the argument

Key machinery is the two-level bandit with sample-replace-predict updates. The outer arm is a word w with global sensitivity G_w^t; the inner arm is the set S_w of sentences containing w, each with a local sensitivity. Selection uses Thompson Sampling (or UCB) over Beta(α, β) priors, so words believed more sensitive are explored more. Local sensitivity is L_w = ε r1 + (1−ε) r2, where r1 and r2 come from a random and the best sampled perturbed sentence and any positive flip is binarized to 1; global sensitivity is updated as G_w^t = (N^w $G_w^{{t−1}}$ + L_w)/(1 + N^w). The machinery's job is to make the exponential enumeration of Hahn et al.'s subset sensitivity unnecessary: SMAB's quoted time complexity is O(T(|X| + |D||V| cost(f))) rather than O(|D| |V|^{|P|} cost(f)).

What would settle it

Take a small dataset, enumerate all MLM substitutions for each word (or compute Hahn et al.'s exact singleton sensitivity on a subset), and compare SMAB's final word rankings to the exact flip rates: if the rank correlation is near zero, or if words above a 0.9 sensitivity threshold flip labels no more often than randomly chosen words, the estimator is not tracking sensitivity.

Watch

Extended reading notes

Core claim

The paper's central claim is that for any sequence classifier and dataset, the sensitivity of individual words can be estimated as the expected frequency with which replacing that word with a contextually plausible substitute flips the classifier's output, and that this can be done at scale by a two-level multi-armed bandit. The outer arms are the dataset's words; each word points to the sentences containing it as inner arms. At each step the bandit selects a word, samples two of its sentences, replaces the word in each using a masked language model (ten draws per sentence), and labels the perturbed sentences with the target classifier. A positive reward is any label flip, binarized to 1, and the local sensitivity is a convex combination of a randomly chosen sentence's reward and the highest-reward sentence's reward; global sensitivity is a running average over visits. The paper reports that on CHECKLIST-style sentiment tests, words marked high-sensitivity by Thompson Sampling lie in DIR templates and flip labels with high success rate, while INV-template words stay in low-sensitivity bins. It further reports that the KL divergence between sensitivity distributions of two models or languages correlates negatively with their accuracy difference (Pearson r = −0.75 on multilingual hate speech, −0.91 on XNLI), and that sensitivity-guided perturbation instructions outperform the perturbation-prompt baselines by 15.58 percentage points on SST-2, while a sensitivity reward improves the paraphrase attack by 12.00 percentage points on human-judged adversarial quality.

Load-bearing premise

The load-bearing premise is that replacing a word in only ten MLM-drawn samples and binarizing every label flip to 1 produces a reward whose average tracks the true probability that perturbing that word changes the classifier's output, so that the bandit's final sensitivity rankings are faithful to the sensitivity it claims to scale up.

Editorial extensions

If this is right

  • Word-level sensitivity can be computed for large datasets without ever seeing model weights or gold labels, using only an MLM and classifier predictions.
  • Template-generated behavioral tests can be audited automatically: words from DIR templates should score high and INV words low, giving a cheap sanity check on a classifier's sensitivities.
  • In the absence of gold labels, the KL divergence between sensitivity distributions can rank models or languages by expected accuracy, which would ease low-resource evaluation.
  • Adversarial attacks can be steered by sensitivity: telling an LLM which words are globally sensitive improves perturbation prompts, and rewarding paraphrase generation for changing sensitivity improves attack quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because SMAB consumes only predictions, the same estimator could audit any API-only classifier; the transferability of sensitivity rankings across different target models is an open question the paper does not test.
  • Beyond the paper: the KLD-accuracy correlation is established on two tasks and a handful of models; a natural test is whether it survives across more diverse model families and domains, and whether it can be used to detect distribution shift.
  • Beyond the paper: the binarization of local rewards to 1 discards confidence information; a variant that keeps the original reward might yield finer-grained sensitivity and could be compared against the current version.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. The paper proposes SMAB, a two-level multi-armed bandit framework for estimating word-level local (sentence-level) and global (dataset-level) sensitivities of a text classifier, following the sensitivity notion of Hahn et al. (2021). The framework is applied to three tasks: a case study on a CheckList-generated sentiment dataset to identify high- and low-sensitivity words, a cross-lingual/cross-model study using KL divergence between sensitivity distributions as an unsupervised accuracy proxy, and two adversarial attack settings (PromptAttack with GPT-3.5 and Llama-2, and ParaphraseAttack on Rotten Tomatoes). The authors report improvements over baselines, including a claimed 15.58% improvement in attack success rate and a 12.00% improvement in human-judged adversariality for paraphrase attacks.

Significance. If the claims held, SMAB would be a practically valuable black-box sensitivity estimator that scales beyond the exponential cost of Hahn et al. (2021), with downstream benefits for model diagnosis, interpretability, and adversarial auditing. The paper contains several constructive elements: a public code repository, a diverse multilingual evaluation setup, human evaluation for paraphrase attacks, and a sanity check on templated CheckList data. However, the significance is conditional on resolving definitional and algorithmic gaps: the estimator as specified does not provably track the sensitivity quantity it claims to scale, the bandit machinery is not fully specified, and one advertised application (paraphrase attack) does not actually use the SMAB estimator. These issues must be addressed before the empirical results can be taken as evidence for the central claim.

major comments (5)
  1. [Section 2.3, Eqs. (2)-(3) and Algorithm 1] The local sensitivity L_w is computed from a randomly sampled sentence s1 and the highest-reward sentence s2 from the perturbed set P_w, and is then binarized ('We assign L_w to 1 if L_w > 0') before the global update in Eq. (3). Since rewards are label-flip indicators, this binarized value is 1 exactly when either s1 or s2 flips the label, so the running average G_w^t converges to the probability that at least one of two (partly cherry-picked) perturbed instances flips the label, not to the mean flip probability that defines singleton sensitivity in Hahn et al. (2021). This quantity is increasing in the number of MLM replacements N and in the number of sentences containing the word, so frequent words are systematically inflated and the resulting ranking is not comparable across words. The paper provides no convergence proof, no regret analysis, and no comparison against exact singleton or block sensitivity on a small dataset, leaving the fidelity of SMAB to the quantity it claims to scale unverified.
  2. [Section 2.3 and Algorithm 1] The described sampling procedure is not a valid UCB or Thompson Sampling implementation. Eq. (1) writes w*_{t+1} = argmax_w Beta(α, β), which is not a well-defined operation, and the Beta parameters α, β are initialized randomly and never updated with observed rewards. Consequently there is no exploration-exploitation tradeoff, no posterior learning, and no mechanism for the global sensitivity estimates to influence future word selection. The statement 'We minimize the total regret R_t' is not supported by any step of the algorithm or by any experimental measurement of regret. As written, SMAB is a random-sampling heuristic with a running-average accumulator, not a multi-armed bandit method.
  3. [Section 5.4 and Appendix D] The paraphrase-attack experiment does not use the SMAB framework. The sensitivity reward in Eq. (9) is computed by Algorithm 2, which extracts keyphrases with TopicRank, masks each word, generates 10 perturbed sentences, and computes the flip fraction; this is a separate estimator with no MAB component and no connection to the local/global sensitivities defined in Section 2.3. Therefore the claimed application of SMAB to paraphrase attacks is not supported by the reported experiments, and the '12.00% improvement over SOTA' attributed to sensitivity-guided paraphrase generation must be re-associated with a different estimator.
  4. [Section 4.3, Figures 3-4] The accuracy-proxy claim rests on only 9 data points for mHate and 5 for XNLI, with no error bars or cross-validation. The p-values treat languages as independent samples, yet languages in the same dataset share model, tokenizer, and task characteristics and are not independent observations. Moreover, Section 4 states that KLD is compared with the 'relative drop in accuracy', but the figures plot KLD against absolute accuracy, which is a different quantity. Given the small sample size and this specification mismatch, the evidence for an unsupervised accuracy proxy is thin.
  5. [Section 5.3, Table 3] The headline attack improvements are reported for the best configuration among several choices of SMAB LLM (six models) and perturbation type (W4, W5, W6), with no standard deviation, confidence interval, or significance test. No comparison is made against a random-word selection baseline or against an alternative attribution method such as LIME or SHAP, so it is unclear whether the observed ASR gains are due to the sensitivity values themselves or to the specific prompt phrasing and the general instruction to modify a minimal subset of words.
minor comments (9)
  1. [Section 2.3 vs Algorithm 1] Step 7 of Algorithm 1 says 'Select two sentences s1, s2 ∈ S_w' while the text says s1 ∈ P_w and s2 ∈ P_w; the notation should be unified to avoid ambiguity about whether the two sentences are drawn from the original sentence set or from the set of perturbed instances.
  2. [Section 2.3] The sentence 'We assign L_w to 1 if L_w > 0' appears in the Global Sensitivity paragraph after Eq. (3), but this binarization is a core part of the estimator and should be defined earlier, with its effect on the statistical interpretation of G_w^t made explicit.
  3. [Abstract vs Section 5.3 and Table 3] The abstract reports an improvement of 13.61% while Section 5.3 and Table 3 report 15.58%; these numbers should be reconciled.
  4. [Section 5.3] The phrase 'improvement over the baselines by a margin of 15.58%' should clarify whether this is an absolute percentage-point difference or a relative improvement, as the two readings give different substantive claims.
  5. [Section 2.3] UCB is mentioned in the text and in Figure 2, but only a Beta-based sampling rule is described, which is not UCB; the paper should either define the UCB rule actually used or state that only Thompson-style sampling was implemented.
  6. [Various] The symbol N is overloaded: it denotes the number of MLM replacements (N=10), the count N^w in Eq. (3), and the number of sensitivity bins in Section 4; using distinct symbols would prevent confusion.
  7. [Appendix A] The time-complexity expression for SMAB in Eq. (8) omits the inner-loop cost of perturbing all sentences in S_w and generating N replacements per selected word; a precise cost model and empirical runtime would be needed to support the scalability claim.
  8. [Figure 2] The SASR curve for UCB is constant because the UCB estimates fall into only two bins, which suggests the UCB variant is not operating as intended; this observation should be discussed rather than used solely as evidence for the superiority of Thompson Sampling.
  9. [Section 3] The sensitivity threshold used in the SASR evaluation is introduced without a principled method for selecting it; the SASR-versus-threshold curves are informative, but the metric requires a clear decision rule for practical use.

Circularity Check

1 steps flagged · score 2.0 of 10

SASR evaluation is self-confirming, but sensitivity-proxy and attack claims are externally validated; no construction-level circularity in the central derivation.

  1. fitted input called prediction [Section 3 'Evaluation' (SASR); Section 2.3 Eqs. (2)-(3), Algorithm 1 lines 7-8]
    "Lw = ϵ ∗ r1 + (1− ϵ) ∗ r2 ... We assign Lw to 1 if Lw > 0 ... Gw_t = (N w ∗ Gw_{t−1} + Lw)/(1 + N w) ... if the word is above the sensitivity threshold and replacing the word with the predictions of a masked language model flips the predicted label in any one of the sentences from Sw, it is called a success. SASR is the fraction of all the words in a dataset above the sensitivity threshold that can flip the predicted label."

    Because r2 is the maximum reward over all sampled MLM replacements, the binarized local sensitivity is 1 iff at least one replacement in the sampled sentences flips the label. Eq. (3) is a running average of that indicator. The SASR 'success' condition is the same replacement-flips-label event on Sw. Thus, for in-sample Sw, SASR is not an independent measure of whether SMAB captured sensitivity; it re-aggregates the very flip signal used to construct G. The threshold-SASR rise is, to that extent, a self-consistency check. The KLD-accuracy proxy and the attack/CHO evaluations use external labels or held-out outcomes, so the circularity is partial and localized to the CHECKLIST quantitative evaluation.

full rationale

SMAB's sensitivity values are produced by a sample-replace-predict MAB procedure; none of the paper's headline validations is fit to the external signal. The KLD-vs-accuracy correlation is computed against gold-label accuracy on the same datasets (an external benchmark), the PromptAttack and ParaphraseAttack results are measured by ASR/CHO against unmodified baselines, and the INV-vs-DIR separation is an external template-structure check. The main circularity-adjacent element is the SASR metric in Section 3, which uses the same replace-and-flip predicate that defines the local sensitivity statistic; this is a consistency metric rather than an independent ground truth. Self-citations (e.g., Ahuja et al. 2022a,b; Das et al. 2024) motivate low-resource evaluation and supply human-evaluation metrics but are not load-bearing for the central derivation. A separate correctness concern, not a circularity one, is that the binarized 'exists-a-flip' average in Eq. (3) has not been shown to converge to Hahn et al.'s singleton sensitivity; this affects fidelity of the estimator to the stated theoretical target but does not make the derivation circular. Overall the derivation chain is mostly self-contained against external benchmarks, with one localized self-confirming evaluation.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The SMAB estimator is an empirical procedure with hand-set hyperparameters and untested modeling assumptions. It does not introduce physical entities, mediators, or forces; the sensitivity scores are summary statistics computed from the classifier and its perturbations, not entities with independent empirical handles outside the paper.

free parameters (7)
  • epsilon (convex weight in local sensitivity) = 0.9
    Convex weight in Eq. 2, set in Section 2.4; after binarization in Eq. 3 its influence on G_w is largely cancelled.
  • Beta prior parameters alpha, beta = alpha random in (0, 0.5), beta = 1 - alpha
    Initial global sensitivities and Thompson sampling draws use Beta(alpha, beta) with alpha drawn afresh (Section 2.4); arbitrary initialization affects exploration.
  • Number of MLM replacements N = 10
    Each selected word is replaced 10 times per pull (Section 2.3); no sensitivity analysis of how N affects estimate accuracy.
  • MAB iterations T = 200,000
    Total bandit steps (Section 2.4); no convergence check or independent stopping criterion.
  • Reward scaling alpha in paraphrase attack = 0.25
    Scale for the sensitivity reward in the paraphrase attack reward (Appendix D.1, Eq. 9); set without tuning analysis.
  • KLD bin count = 10
    Sensitivity distributions are binned into 10 bins before KL divergence (Section 4); correlation values may depend on this binning.
  • SASR sensitivity threshold = grid from 0 to 0.9
    SASR is reported across thresholds (Section 3, Figure 2); the claim that high-sensitivity words flip labels depends on the threshold choice.
assumptions (5)
  • domain assumption Singleton word replacements faithfully approximate subset sensitivity
    Local sensitivity is computed with singleton replacements (Section 2.2 footnote) while Hahn et al. sensitivity counts disjoint subsets; no exact comparison is run.
  • domain assumption MLM replacements are semantically coherent and label-preserving apart from the target word
    The sample-replace-predict loop (Section 2.3) assumes masked-language-model substitutions are valid perturbations rather than artifacts that flip labels for unrelated reasons.
  • ad hoc to paper The binarized running average in Eq. 3 converges to the true global sensitivity
    G_w is an average of indicators Lw>0; no proof or regret bound is given that this recovers the minimal-subset heuristic defined in Section 2.2.
  • domain assumption Ten-bin KL divergence captures accuracy-relevant distribution shift
    The proxy-for-accuracy analysis (Section 4) uses 10 equal bins and DKL without justifying the binning or the direction of the divergence.
  • domain assumption The small language and model samples are representative
    Correlations use 9 (mHate) and 5 (XNLI) points (Section 4.3); no analysis of confounders such as language family, dataset size, or task difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation." pith.science (2026). https://pith.science/paper/SCYTOS6R

@misc{pith2026250207101,
  author       = {Pith},
  title        = {Pith review of: SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SCYTOS6R}},
  note         = {Machine review of arXiv:2502.07101}
}
read the original abstract

To understand the complexity of sequence classification tasks, Hahn et al. (2021) proposed sensitivity as the number of disjoint subsets of the input sequence that can each be individually changed to change the output. Though effective, calculating sensitivity at scale using this framework is costly because of exponential time complexity. Therefore, we introduce a Sensitivity-based Multi-Armed Bandit framework (SMAB), which provides a scalable approach for calculating word-level local (sentence-level) and global (aggregated) sensitivities concerning an underlying text classifier for any dataset. We establish the effectiveness of our approach through various applications. We perform a case study on CHECKLIST generated sentiment analysis dataset where we show that our algorithm indeed captures intuitively high and low-sensitive words. Through experiments on multiple tasks and languages, we show that sensitivity can serve as a proxy for accuracy in the absence of gold data. Lastly, we show that guiding perturbation prompts using sensitivity values in adversarial example generation improves attack success rate by 15.58%, whereas using sensitivity as an additional reward in adversarial paraphrase generation gives a 12.00% improvement over SOTA approaches. Warning: Contains potentially offensive content.

Figures

Figures reproduced from arXiv: 2502.07101 by the authors.

Figure 1
Figure 1. Overview of our SMAB framework. The outer arm consists of all words in the corpus, each linked to a set of sentences in the inner arm. w1 is a word in the outer arm, and Sw1 is the set of sentences in the inner arm that contains w1. G w1 t is the Global Sensitivity of word w1 at step t. We utilize a sample-replace-predict strategy to estimate local sensitivity values Lw1 for a word w1. Here, Pw1 is the set of predic… view at source ↗
Figure 2
Figure 2. Variation of SASR with sensitivity threshold [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. KL Divergence v/s accuracy across languages [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: KL Divergence v/s accuracy across languages [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Scatter plot of estimated global sensitivities of [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: KLD v/s accuracy within language on mHate [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 8 canonical work pages

  1. [1]

    Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022 a . Beyond static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages. arXiv preprint arXiv:2205.06356

    work page Pith review arXiv 2022

  2. [2]

    Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022 b . Multi task learning for zero shot performance prediction of multilingual models. arXiv preprint arXiv:2205.06130

    work page Pith review arXiv 2022

  3. [3]

    Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235--256

    2002

  4. [4]

    Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. https://doi.org/10.18653/v1/S19-2007 S em E val-2019 task 5: Multilingual detection of hate speech against immigrants and women in T witter . In Proceedings of the 13th International Workshop on Semant...

    doi:10.18653/v1/s19-2007 2019

  5. [5]

    Mohit Bhardwaj, Md Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. 2020. https://arxiv.org/abs/2011.03588 Hostility detection dataset in hindi . Preprint, arXiv:2011.03588

    arXiv 2020

  6. [6]

    Florian Boudin. 2016. http://aclweb.org/anthology/C16-2015 pke: an open source python-based keyphrase extraction toolkit . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 69--73, Osaka, Japan

    work page 2016

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  8. [8]

    Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020. https://doi.org/10.1609/aaai.v34i04.5767 Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples . Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3601--3608

    doi:10.1609/aaai.v34i04.5767 2020

Show all 41 references
  1. [9]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2022 arXiv

  2. [10]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . Prepri...

    2020 arXiv

  3. [11]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  4. [12]

    Mithun Das, Saurabh Pandey, Shivansh Sethi, Punyajoy Saha, and Animesh Mukherjee. 2024. https://aclanthology.org/2024.findings-eacl.111 Low-resource counterspeech generation for I ndic languages: The case of B engali and H indi . In Findings of the Association for Computationa...

    2024

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    2019 arXiv

  6. [14]

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. https://doi.org/10.18653/v1/P18-2006 H ot F lip: White-box adversarial examples for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa...

    2018 doi

  7. [15]

    Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.29 Latent hatred: A benchmark for understanding implicit hate speech . In Proceedings of the 2021 Conference ...

    2021 doi

  8. [16]

    Michael Hahn, Dan Jurafsky, and Richard Futrell. 2021. https://doi.org/10.1162/tacl_a_00403 Sensitivity as a complexity measure for sequence classification tasks . Transactions of the Association for Computational Linguistics, 9:891--908

    2021 doi

  9. [17]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    2021 arXiv

  10. [18]

    Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1170 Adversarial example generation with syntactically controlled paraphrase networks . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association ...

    2018 doi

  11. [19]

    Sheng Lu, Hendrik Schuff, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.naacl-long.325 How are prompts different in terms of sensitivity? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2024 doi

  12. [20]

    Scott Lundberg and Su-In Lee. 2017. https://arxiv.org/abs/1705.07874 A unified approach to interpreting model predictions . Preprint, arXiv:1705.07874

    2017 arXiv

  13. [21]

    Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. https://doi.org/10.1145/3368567.3368584 Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages . In...

    2019

  14. [22]

    Bo Pang and Lillian Lee. 2005. https://doi.org/10.3115/1219840.1219855 Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics ( ACL ' 0...

    2005

  15. [23]

    Juan Carlos Pereira-Kohatsu, Lara Quijano S \'a nchez, Federico Liberatore, and Miguel Camacho-Collados. 2019. https://api.semanticscholar.org/CorpusID:207961644 Detecting and monitoring hate speech in twitter . Sensors (Basel, Switzerland), 19

    2019

  16. [24]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. https://arxiv.org/abs/1602.04938 "why should i trust you?": Explaining the predictions of any classifier . Preprint, arXiv:1602.04938

    2016 arXiv

  17. [25]

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. https://doi.org/10.18653/v1/P18-1079 Semantically equivalent adversarial rules for debugging NLP models . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2018 doi

  18. [26]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020 doi

  19. [27]

    Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, and Md Saiful Islam. 2020. https://arxiv.org/abs/2012.09686 Hate speech detection in the bengali language: A dataset and its baseline evaluation . Preprint, arXiv:2012.09686

    2020 arXiv

  20. [28]

    Tom Roth, Inigo Jauregi Unanue, Alsharif Abuadbba, and Massimo Piccardi. 2024. https://arxiv.org/abs/2405.11904 A constraint-enforcing reward for adversarial attacks on text classifiers . Preprint, arXiv:2405.11904

    2024 arXiv

  21. [29]

    V Sanh. 2019. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

    2019 arXiv

  22. [30]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://openreview.net/forum?id=RIu5lyNXjT Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . In The Twelfth International...

    2024

  23. [31]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language...

    2013

  24. [32]

    William R Thompson. 1933. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285--294

    1933

  25. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  26. [34]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. https://doi.org/10.18653/v1/D19-1221 Universal adversarial triggers for attacking and analyzing NLP . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the...

    2019 doi

  27. [35]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  28. [36]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  29. [37]

    Williams

    Ronald J. Williams. 2004. https://api.semanticscholar.org/CorpusID:2332513 Simple statistical gradient-following algorithms for connectionist reinforcement learning . Machine Learning, 8:229--256

    2004

  30. [38]

    Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. 2023. https://arxiv.org/abs/2310.13345 An llm can fool itself: A prompt-based adversarial attack . Preprint, arXiv:2310.13345

    2023 arXiv

  31. [39]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.