Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Text classifiers can be certified against word and character substitutions in two forward passes by enclosing all reachable altered sentences in a simplex and propagating interval bounds through the network.

desk verdict Simplex + IBP gives a real, efficient verification route for text classifiers, but the paper must state and enforce a single-token synonym filter for the certificate to cover the attack set. read the letter →

arxiv 1909.01492 v2 pith:HLRUAN2J submitted 2019-09-03 cs.CL cs.CRcs.LGstat.ML

Po-Sen Huang , Robert Stanforth , Johannes Welbl , Chris Dyer , Dani Yogatama , Sven Gowal , Krishnamurthy Dvijotham , Pushmeet Kohli This is my paper · ORCID
classification cs.CLcs.CRcs.LGstat.ML
keywords verifiedrobustnessintervalboundpropagationtextclassificationsymbolsubstitutionattacksadversarialneuralnetworkverificationcertifiedtrainingwordandcharacterperturbations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that formal verification, rather than adversarial training or data augmentation, can give text classifiers a genuine worst-case guarantee against discrete symbol substitutions. It represents every sentence reachable by up to $\delta$ synonym swaps or character flips as a point inside a simplex in embedding space, then uses Interval Bound Propagation to compute a sound upper bound on the largest possible violation of the true-class-wins specification. Because that bound costs only two forward passes, it can also serve as a training objective, producing models whose certified accuracy on SST and AG News is close to their nominal accuracy. The authors use exhaustive enumeration on small perturbation radii to confirm that the certificates are meaningful, and find that adversarial training, despite high accuracy against HotFlip attacks, is much less robust when every possible substitution is checked.

What carries the argument

The load-bearing object is the input simplex $\bar{X}_{in}(x_0) = \mathrm{conv}\{x_0, x_0+\delta(p_0^{(1)}-x_0), \dots, x_0+\delta(p_0^{(M)}-x_0)\}$, where each $p_0^{(m)}$ is the embedding of a sentence that differs from $x_0$ by one allowed substitution. This set over-approximates the true attack space but, unlike an axis-aligned box, does not include sentences with all positions substituted at once, making it much tighter for small $\delta$. Interval Bound Propagation then bounds each layer's activations by interval arithmetic, starting with the exact extrema of the first affine layer over the $M+1$ vertices and propagating axis-parallel boxes through subsequent affine and monotonic layers. The resulting upper bound on the worst-case logit difference is used both as a verification certificate and, through the loss $\mathcal{L} = \kappa\mathcal{L}_{\mathrm{normal}} + (1-\kappa)\mathcal{L}_{\mathrm{spec}}$, as a training signal that makes the certificate tight.

What would settle it

Take a concrete test sentence and a valid perturbation of exactly two distinct positions, compute the embedding of that perturbed sentence, and check whether it lies inside the simplex by solving the linear feasibility problem for convex-combination coefficients. Finding one valid two-substitution sentence whose embedding falls outside the hull would refute the containment assumption; checking hundreds of random such sentences and finding none would support it.

Watch

Extended reading notes

Core claim

The central claim is that discrete substitution attacks have a tractable convex surrogate: the convex hull of the original sentence embedding and of the $\delta$-scaled single-substitution embeddings contains every sentence obtainable by up to $\delta$ substitutions. Interval Bound Propagation over this simplex yields an upper bound on the worst-case logit difference, and training against that bound closes most of the gap between the certificate and the exact oracle. In the paper's experiments, verifiably trained models achieve the highest exhaustive verification accuracy on both datasets: 73.1% versus 25.8% for adversarial training on SST character-level at $\delta=3$, with only a 5.6-point drop in nominal accuracy. The paper also demonstrates that heuristic attack accuracy can be a false comfort: adversarial training looks strong under HotFlip but collapses under exhaustive verification.

Load-bearing premise

The entire guarantee rests on the containment claim that the convex hull of the original embedding and the $\delta$-scaled single-substitution vertices contains every sentence reachable by up to $\delta$ substitutions, with fixed non-contextual embeddings; if a valid perturbed sentence falls outside that simplex, the IBP certificate no longer covers the true attack set.

Editorial extensions

If this is right

  • On SST character-level at $\delta=3$, verifiable training raises exhaustively verified accuracy from 10.3% (normal training) and 25.8% (adversarial training) to 73.1%, with nominal accuracy falling only from 79.8% to 74.2%.
  • IBP verification costs two forward passes per test sentence, while exhaustive verification of a single SST character-level sentence at $\delta=3$ can require up to 1,436,026 forward passes; the same two forward passes handle any perturbation budget at test time.
  • Adversarial accuracy under HotFlip is not a reliable proxy for true robustness; the paper's oracle results show that models can be robust to the attack yet fail on other valid substitutions from the same threat model.
  • Reducing the simplex volume helps: counter-fitted embeddings, which pull synonyms closer together, improve IBP-verified accuracy by up to 33.2% under normal training at $\delta=1$, and verifiable training benefits further.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the certificate is only as strong as the allowed substitution list; paraphrases, insertions, deletions, or semantic-preserving rewrites that are not in the word or character substitution set fall outside the simplex, so the formal guarantee does not extend to them without redefining the threat model.
  • Editorial extension: the vertex-difference construction presupposes fixed, non-contextual embeddings; in a contextual model, replacing one token changes every position's representation, so the same simplex argument would need to be replaced by a bound over a set of contextual representations.
  • The paper itself notes the method is demonstrated on shallow convolutional and fully-connected models, and that IBP bounds are likely to loosen with depth; extending verifiable training to deep transformers or recurrent networks is a natural next step that this work does not test.
  • Testable extension: for small $\delta$ where exhaustive enumeration remains feasible, one could quantify how conservative the simplex is by computing, for the same models, the gap between the IBP certificate and the exact oracle certificate, and by comparing the simplex with the true convex hull of all $\delta$-way substitutions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a formal verification method for text classifiers under bounded symbol substitutions (synonym replacements or character flips). The authors model the set of allowed perturbed sentences as the convex hull of the original representation and a set of elementary single-substitution vectors, scaled by the perturbation budget δ, and then use Interval Bound Propagation to compute sound upper bounds on the worst-case logit difference over this simplex. They introduce a verifiable training objective that optimizes this bound and evaluate on SST and AG News with character- and word-level perturbations. The key claims are that (i) the simplex over-approximation enables efficient formal verification, (ii) verifiably trained models achieve much higher exhaustively verified (Oracle) accuracy than adversarial training or data augmentation, and (iii) adversarial training can give a false sense of security, as the Oracle accuracy is often far lower than accuracy against HotFlip attacks.

Significance. If the specification issues are resolved, this is a solid and timely contribution. The paper is, to my knowledge, among the first to apply incomplete verification (IBP) to discrete NLP perturbations, and the simplex over-approximation is a neat way to avoid the loose axis-aligned bounding box that would arise from an L∞-style relaxation. The use of an exhaustive verification oracle as a benchmark is a strength: it provides ground-truth robustness numbers that many NLP verification papers lack, and the demonstration that HotFlip-based adversarial training is much weaker under exhaustive verification is an important cautionary result. The experiments with counter-fitted embeddings are a sensible analysis of how embedding geometry affects bound tightness. The code is publicly available. However, the formal guarantee's validity for the actually evaluated attack set is not fully established, and the main table does not report the strength of the efficient IBP certificate, which in some settings is considerably weaker than the Oracle.

major comments (3)
  1. [§3.3 and §4.3] The over-approximation property asserted in Section 3.3 is correct only if every elementary perturbation p^(m)_0 differs from x0 at exactly one position and if multi-substitutions occur at distinct positions. No proof of this containment is given in the paper. More importantly, Section 4.3 constructs word-level synonym pairs from PPDB and filters only by POS tags using Spacy; PPDB contains multiword paraphrases. If such a paraphrase is used, the resulting representation does not differ from x0 at a single position, the simplex vertices are not defined as stated, and the IBP certificate does not cover that perturbation. The paper must explicitly state whether multiword entries were excluded, provide the exact filter, and, if they were not excluded, clearly delineate that the formal guarantee applies only to the single-token subset. Otherwise the reported Oracle and adversarial accuracies are not measuring the set for which the formal guarantee holds.
  2. [§4.5, Table 1, Figures 2–3] Table 1 reports Oracle (exhaustively verified) accuracy for the verifiable models but omits the IBP-verified accuracy, which is the quantity actually certified by the efficient method. The figures show that for word-level models the IBP-verified accuracy is substantially below the Oracle accuracy; for example, in Figure 3b at δ=3 the Oracle value appears to be near 76.5% while the IBP-verified value appears to be near 44.5%. Because the abstract and conclusion emphasize an 'efficiently computable formal guarantee,' the main table should report IBP-verified accuracy alongside the Oracle, or the text should clearly state that the efficient certificate is weaker than the Oracle numbers suggest. As written, a reader could infer that the efficient method certifies all examples counted by the Oracle figure.
  3. [§3.4 (Eq. 4), §4.5, Appendix B.3] Equation (4) requires computing the first layer's output for each of the M scaled vertices, so the verification cost grows with M, the number of elementary perturbations. The statements that 'IBP verification only requires 2 forward passes' (Appendix B.3) and 'constant-time verification' (Section 4.5) are not supported by the method as described. The authors should either provide a closed-form way to compute the first-layer bounds without enumerating all M vertices, or qualify the claim, for example as 'two forward passes through the network after computing first-layer bounds over the M elementary perturbations.' This is important because the efficiency of verification is a central contribution.
minor comments (5)
  1. [§4.3] Please specify the exact criterion used to select PPDB entries for word-level experiments, including whether single-token and single-token-only replacements were enforced, and report the resulting number of synonym pairs per vocabulary or per sentence.
  2. [§4.5 and Table 1] The main results would be more reliable if the experiments were repeated over multiple seeds and reported with mean and standard deviation; the current single-run numbers make it difficult to judge the significance of the observed differences.
  3. [§3.4, Eq. (4)] The symbol e_i should be defined as the standard basis vector for coordinate i, and the notation zK,ytrue in Eq. (2) and Eq. (7) should be typeset consistently (e.g., zK,y_true).
  4. [Figures 2–4] The term 'Oracle' is used for 'exhaustively verified accuracy'; please define it once in the text and avoid potential confusion with a ground-truth label oracle.
  5. [Figure 4] The dashed horizontal lines for IBP verification should be described with respect to their computational cost on the x-axis; if the first-layer enumeration is counted, the lines should not be at a fixed low x-coordinate.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the IBP guarantee is an independently checkable bound, with only minor self-citation to prior IBP work.

full rationale

The paper's verification claim is not circular: the attack set X_in is defined independently as all sentences reachable by up to delta synonym or character substitutions, the simplex S_delta is a convex hull of scaled single-substitution vertices, and the containment S_delta superset of X_in is a mathematical claim that can be checked independently of the model's outputs. IBP bounds are then computed from the network and the input region via interval arithmetic, and the paper evaluates them against an exhaustive oracle that enumerates the true attack set; the oracle agreement is external evidence, not a fitted target. The IBP training loss (Eq. 7) does optimize the same upper bound used at test time, but this is a training-objective choice, not a circular prediction: the reported IBP-verified accuracy is a sound lower bound only if the over-approximation is sound, and the oracle numbers independently confirm the ordering. The paper cites prior IBP work by overlapping authors (Gowal et al., 2018; Dvijotham et al., 2018), but the relevant bound-propagation equations are restated in Section 3.4 and the approach is benchmarked against exhaustive verification, so the self-citation is not load-bearing in a way that forces the result. The unproved simplex containment assertion in Section 3.3 is a soundness and correctness risk, especially if PPDB multiword paraphrases entered the word-level attack set without an explicit single-token filter, but that is not circularity: the definitions do not presuppose the conclusion.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method introduces no new physical or mathematical entities. It does depend on standard IBP soundness, a domain assumption about fixed embeddings, a plausible but under-proved simplex containment claim, and a chosen threat model. The free parameters are training hyperparameters, not fitted constants inside the verification bound, so they do not make the certificate circular.

free parameters (4)
  • Loss interpolation weight kappa = Curriculum from 1 to 0.25 (final)
    Controls the trade-off between normal cross-entropy loss and the IBP specification loss in Eq. (7); tuned on validation and central to the verifiable training procedure.
  • Training perturbation budget delta = 3
    The maximum number of symbol substitutions used during training; models are evaluated at delta from 1 to 6, so this choice affects the certificate tightness.
  • Adversarial and data augmentation interpolation weight = 0.5
    Baseline methods mix the original loss and the adversarial or augmentation loss with weight 0.5; this is a chosen hyperparameter with no sensitivity analysis.
  • Model architecture hyperparameters = 100 filters, kernel widths 5 or 10, embedding dims 150 or 300
    Selected to give a reasonable accuracy-robustness trade-off; these choices affect the reported numbers but not the method's validity.
assumptions (5)
  • standard math IBP interval arithmetic gives valid upper and lower bounds on network outputs for monotonic activations and affine layers
    The paper relies on this from Gowal et al. (2018) and Mirman et al. (2018) without reproving it; it is the foundation of the certificate.
  • domain assumption The convex hull of the dilated elementary substitutions contains all sentences with up to delta substitutions
    Stated in Section 3.3 with only a short justification and no formal proof; if this containment fails, the certificate does not cover the true attack set.
  • domain assumption Input perturbations correspond to replacing fixed embedding vectors at distinct positions, with no contextual embeddings
    Section 3.3 explicitly says contextual embeddings are not considered, so the simplex is defined in a fixed embedding space and the guarantee applies only to that representation.
  • domain assumption The first layer is affine followed by a monotonic activation, so Eq. (4) computes exact bounds by evaluating the M+1 simplex vertices
    The models use CNN plus ReLU, which satisfies this, but the method is only sound for such architectures.
  • domain assumption The PPDB synonym set filtered by POS tags and the keyboard adjacency typo set define the threat model
    The formal guarantee covers only these predefined perturbations, not all semantically invariant attacks such as paraphrases or insertions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation." pith.science (2026). https://pith.science/paper/HLRUAN2J

@misc{pith2026190901492,
  author       = {Pith},
  title        = {Pith review of: Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HLRUAN2J}},
  note         = {Machine review of arXiv:1909.01492}
}
read the original abstract

Neural networks are part of many contemporary NLP systems, yet their empirical successes come at the price of vulnerability to adversarial attacks. Previous work has used adversarial training and data augmentation to partially mitigate such brittleness, but these are unlikely to find worst-case adversaries due to the complexity of the search space arising from discrete text perturbations. In this work, we approach the problem from the opposite direction: to formally verify a system's robustness against a predefined class of adversarial attacks. We study text classification under synonym replacements or character flip perturbations. We propose modeling these input perturbations as a simplex and then using Interval Bound Propagation -- a formal model verification method. We modify the conventional log-likelihood training objective to train models that can be efficiently verified, which would otherwise come with exponential search complexity. The resulting models show only little difference in terms of nominal accuracy, but have much improved verified accuracy under perturbations and come with an efficiently computable formal guarantee on worst case adversaries.

Figures

Figures reproduced from arXiv: 1909.01492 by the authors.

Figure 1
Figure 1. Illustration of verification with the input [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SST character-level models with different training objectives (trained at [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. SST word-level models with normal and verifiable training objectives (trained at [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Verified accuracy vs. computation budget [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: SST word-level models with different training objectives (trained at [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: SST word-level models (trained at δ=3) using counter-fitted embeddings against different perturba￾tion budgets in nominal accuracy, adversarial accuracy, exhaustively verified accuracy (Oracle), and IBP verified accuracy. Note that exhaustive verification is not scalab…
Figure 7
Figure 7. Figure 7: AG News character-level models with different training objectives (trained at [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anish Athalye, Nicholas Carlini, and David A. Wagner. 2018. http://proceedings.mlr.press/v80/athalye18a.html Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples . In ICML, pages 274--283

    work page 2018

  4. [4]

    Christel Baier and Joost-Pieter Katoen. 2008. Principles of Model Checking. MIT press

    work page 2008

  5. [5]

    Clark Barrett and Cesare Tinelli. 2018. Satisfiability modulo theories. In Handbook of Model Checking, pages 305--343. Springer

    work page 2018

  6. [6]

    Yonatan Belinkov and Yonatan Bisk. 2018. https://openreview.net/forum?id=BJ8vJebC- Synthetic and natural noise both break neural machine translation . In International Conference on Learning Representations

    2018

  7. [7]

    Nicola Bertoldi, Richard Zens, and Marcello Federico. 2007. Speech translation by confusion network decoding. In Proc. ICASSP

    work page 2007

  8. [8]

    Rudy Bunel, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar. 2017. Piecewise linear neural network verification: a comparative study. arXiv preprint arXiv:1711.00455

    work page Pith review arXiv 2017

Show all 47 references
  1. [9]

    Nicholas Carlini, Guy Katz, Clark Barrett, and David L Dill. 2017. Ground-truth adversarial examples. arXiv preprint arXiv:1709.10207

    2017 arXiv

  2. [10]

    Nicholas Carlini and David Wagner. 2017. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3--14. ACM

    2017

  3. [11]

    Chih-Hong Cheng, Georg N \"u hrenberg, and Harald Ruess. 2017. Maximum resilience of artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis, pages 251--268. Springer

    2017

  4. [12]

    Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin - Yu Chen, and Cho - Jui Hsieh. 2018. Seq2Sick : Evaluating the robustness of sequence-to-sequence models with adversarial examples. CoRR, abs/1803.01128

    2018 arXiv

  5. [13]

    Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, and Pushmeet Kohli. 2018. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265

    2018 arXiv

  6. [14]

    Christopher Dyer, Smaranda Muresan, and Philip Resnik. 2008. https://www.aclweb.org/anthology/P08-1115 Generalizing word lattice translation . In Proceedings of ACL-08: HLT, pages 1012--1020, Columbus, Ohio. Association for Computational Linguistics

    2008

  7. [15]

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. http://aclweb.org/anthology/P18-2006 HotFlip : White-box adversarial examples for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pape...

    2018

  8. [16]

    Allyson Ettinger, Sudha Rao, Hal Daum \'e III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems

    2017

  9. [17]

    Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. https://www.aclweb.org/anthology/N13-1092 PPDB : The paraphrase database . In Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langua...

    2013

  10. [18]

    Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. https://doi.org/10.18653/v1/P18-2103 Breaking NLI systems with sentences that require simple lexical inferences . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short P...

    2018 doi

  11. [19]

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572

    2014 arXiv

  12. [20]

    Mann, and Pushmeet Kohli

    Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy A. Mann, and Pushmeet Kohli. 2018. http://arxiv.org/abs/1810.12715 On the effectiveness of interval bound propagation for training verifiably robust mod...

    2018 arXiv

  13. [21]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  14. [22]

    Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1170 Adversarial example generation with syntactically controlled paraphrase networks . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association ...

    2018 doi

  15. [23]

    Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP)

    2017

  16. [24]

    Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pages 97--117. Springer

    2017

  17. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations

    2015

  18. [26]

    Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533

    2016 arXiv

  19. [27]

    Yitong Li, Trevor Cohn, and Timothy Baldwin. 2017. Robust training under linguistic adversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 21--27

    2017

  20. [28]

    Xunying Liu, Xie Chen, Yongqiang Wang, Mark J. F. Gales, and Philip C. Woodland. 2016. Two efficient lattice rescoring methods using recurrent neural network language models. IEEE/ACM Trans. Audio, Speech & Language Processing , 24(8):1438--1449

    2016

  21. [29]

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. https://openreview.net/forum?id=rJzIBfZAb Towards deep learning models resistant to adversarial attacks . In International Conference on Learning Representations

    2018

  22. [30]

    George A. Miller. 1995. WordNet : A lexical database for English . Commun. ACM, 38(11):39--41

    1995

  23. [31]

    Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable abstract interpretation for provably robust neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3578--3586

    2018

  24. [32]

    Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young

    Nikola Mrk s i \'c , Diarmuid \'O S \'e aghdha, Blaise Thomson, Milica Ga s i \'c , Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. https://doi.org/10.18653/v1/N16-1018 Counter-fitting word vectors to linguistic constraints . In Proce...

    2016 doi

  25. [33]

    Takashi Onishi, Masao Utiyama, and Eiichiro Sumita. 2010. https://www.aclweb.org/anthology/P10-2001 Paraphrase lattice for statistical machine translation . In Proceedings of the ACL 2010 Conference Short Papers , pages 1--5, Uppsala, Sweden. Association for Computational Linguistics

    2010

  26. [34]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 GloVe : Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543. Associat...

    2014 doi

  27. [35]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Assoc...

    2018 doi

  28. [36]

    Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified defenses against adversarial examples. In International Conference on Learning Representations

    2018

  29. [37]

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. http://aclweb.org/anthology/P18-1079 Semantically equivalent adversarial rules for debugging NLP models . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2018

  30. [38]

    Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. https://doi.org/10.18653/v1/P18-1041 Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms ....

    2018 doi

  31. [39]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. http://aclweb.org/anthology/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on ...

    2013

  32. [40]

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199

    2013 arXiv

  33. [41]

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In International Conference on Learning Representations

    2019

  34. [42]

    Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aäron van den Oord. 2018. http://proceedings.mlr.press/v80/uesato18a.html Adversarial risk and the dangers of evaluating against weak attacks . In ICML, pages 5032--5041

    2018

  35. [43]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal trigger sequences for attacking and analyzing NLP . In Empirical Methods in Natural Language Processing (EMNLP)

    2019

  36. [44]

    Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. https://www.usenix.org/conference/usenixsecurity18/presentation/wang-shiqi Formal security analysis of neural networks using symbolic intervals . In 27th USENIX Security Symposium ( USENIX Security 1...

    2018

  37. [45]

    Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. 2018. http://proceedings.mlr.press/v80/weng18a.html Towards fast computation of certified robustness for R e LU networks . In Proceedings of the 35th International Co...

    2018

  38. [46]

    Eric Wong and Zico Kolter. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5283--5292

    2018

  39. [47]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649--657

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.