Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Adversarial Demonstration Learning for Low-resource NER Using Dual Similarity

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper claims low-resource NER improves when demonstrations match entity types and the tagger is forced to use them.

desk verdict Useful few-shot NER method with a serious missing-detail threat: the input may be its own demonstration, which would undermine the adversarial-training claim. read the letter →

arxiv 2507.15864 v1 pith:UGPRK2F3 submitted 2025-07-13 cs.CL cs.LG

classification cs.CLcs.LG
keywords namedentityrecognitionlow-resourceNERdemonstrationlearningfew-shotdualsimilarityfeatureJaccardadversarialtrainingStructShot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets few-shot named entity recognition (NER), where only $k$ labeled examples per entity type are available. It argues that demonstration learning underperforms for two fixable reasons: demonstrations are chosen mainly by surface text similarity, ignoring whether example and input contain the same entity types, and the tagger largely ignores the demonstrations it is given. ADELL addresses both problems: it scores candidate examples by dual similarity, a weighted combination of semantic cosine similarity and predicted feature Jaccard similarity, and it trains the StructShot tagger with adversarial demonstration learning, in which example order and label-to-entity mappings are permuted so the model must read the demonstrations to label correctly. On CoNLL03, a German legal corpus, and MIT restaurant reviews, the paper reports F1 gains of roughly 2 to 6 points over dynamic-demonstration baselines, with ablations attributing the gain to both components.

What carries the argument

The load-bearing object is the dual similarity function $S(d_i,d_j)=\gamma\cdot S_{fe}(d_i,d_j)+(1-\gamma)\cdot S_{se}(d_i,d_j)$, where $S_{se}$ is cosine similarity of sentence embeddings and $S_{fe}$ is a cross-encoder's estimate of feature Jaccard similarity, $FJ=|F_a\cap F_b|/|F_a\cup F_b|$. This measure carries demonstration selection: candidates are ranked by $S$ and filtered per entity type so each chosen example is both textually close and entity-type-overlapping with the input. The second mechanism is adversarial demonstration learning: label permutation and example permutation during training create the losses $\ell_e$ and $\ell_l$ that force the language model to attend to the demonstration, combined as $\text{loss}=\alpha\,\ell_m+(1-\alpha)((1-\beta)\ell_e+\beta\,\ell_l)$, with $\alpha$ and $\beta$ chosen by grid search. The underlying tagger is StructShot, whose token classifier and Viterbi decoder turn token-level labels into entity spans.

What would settle it

Construct held-out input/example pairs whose features come only from the $k$-shot training set, compute the feature-similarity predictor's binary and ranking accuracy against exact feature Jaccard similarity, and compare with its reported accuracies on full-distribution pairs; if accuracy falls to near chance on the few-shot-native pairs, the dual-similarity ranking in Eq. (1) loses its advantage and ADELL would be expected to match the semantic-only ablation.

Watch

Extended reading notes

Core claim

ADELL is a demonstration-learning pipeline for low-resource NER. Given an input, its demonstration incorporator ranks, for each entity type, the labeled examples in a demo pool by $S(d_i,d_j)=\gamma\cdot S_{fe}(d_i,d_j)+(1-\gamma)\cdot S_{se}(d_i,d_j)$, discards the half ranked lowest, and samples one example per entity type to form the demonstration; $S_{se}$ is cosine similarity of sentence embeddings and $S_{fe}$ is produced by a cross-encoder trained to predict the Jaccard similarity of the entity-type sets of the two texts. The tagger is StructShot, a nearest-neighbor-style token classifier with a Viterbi decoder for coherent label sequences. During training, ADELL permutes the order of the demonstration examples and swaps entity labels inside them, so the classifier can only label correctly by following the annotation rule shown in the demonstration; the final loss is a weighted combination of the main-task loss, the example-permutation loss, and the label-permutation loss. At application time, $k$ demonstrated inputs are formed and their predictions are majority-voted. The paper reports that ADELL achieves the best F1 among all compared methods on CoNLL03, German legal, and MIT datasets at 5-, 10-, and 20-shot settings, and that both dual-similarity selection and adversarial demonstration learning contribute in ablations.

Load-bearing premise

The feature-similarity predictor, trained on only the few-shot training set, stays accurate enough to rank demonstration examples by entity-type overlap at deployment; the paper's Section 4.4 evaluation draws its pairs from the full training set rather than from the few-shot regime.

Editorial extensions

If this is right

  • Semantic-only demonstration retrieval leaves measurable F1 on the table; combining semantic and feature similarity becomes the stronger default for low-resource NER.
  • A tagger trained with label-permuted demonstrations becomes measurably more attentive: swapping entity labels in test-time demonstrations shifts its label scores, whereas a normally trained tagger is barely affected.
  • Because both ablated versions still beat the dynamic-demonstration baseline, each mechanism is a usable improvement by itself, and the two complement each other.
  • The largest reported gain appears in the most complex setting (German legal at 5-shot: F1 from 35.73 to 41.92), suggesting the method is most useful where demonstrations matter most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dual-similarity recipe should transfer to other structured-prediction tasks where demonstrations carry output labels, such as slot filling or event extraction, by substituting the relevant label set for entity types.
  • The adversarial-attention diagnostic is a reusable tool: measuring how much test-time label swaps move predictions yields a cheap, model-agnostic score of how much a system relies on its demonstrations.
  • A cheaper alternative to training a separate feature-similarity predictor would be to derive entity-type overlap estimates from the tagger's own label distributions; if such self-supervised estimates track the cross-encoder's, the extra component might be removable.
  • Because the feature-similarity predictor is validated on pairs from the full training distribution, its accuracy in the true $k$-shot regime remains the main uncertainty; a few-shot-native evaluation would settle how much of the reported gain survives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses low-resource named entity recognition with demonstration learning. It proposes ADELL, which combines (i) dual similarity for demonstration selection, blending semantic similarity with a learned feature-similarity predictor that estimates the Jaccard similarity of entity-type sets, and (ii) adversarial demonstration learning (ADL), which perturbs example order and label assignments during training to force the StructShot tagger to attend to demonstrations. Experiments on CoNLL03, a German legal dataset, and MIT restaurant reviews under 5/10/20-shot settings report F1 scores over 20 runs, comparing against 12 baseline methods plus two ablations. The paper claims that ADELL achieves the best performance and that both proposed components contribute, with gains of roughly 2–6 F1 points over dynamic demonstration baselines.

Significance. If the results hold, the paper makes a useful contribution to few-shot NER: it identifies a concrete failure mode of semantic-only demonstration selection, proposes a feature-similarity predictor that is complementary to semantic similarity, and introduces an adversarial training scheme that demonstrably increases attention to demonstrations. The experimental protocol is relatively thorough—20 runs with standard deviations on three datasets, ablations for both components, and an attention-based analysis of ADL. The central empirical claims are, however, conditional on resolving several load-bearing issues: the possibility of a demonstration pool including the input itself, the mismatch between predictor evaluation and the actual few-shot setting, and an internal contradiction in Table 1 for the MIT dataset at k=20.

major comments (3)
  1. [§3.1–3.2, Eq. (1)] The manuscript never states that the demonstration pool C excludes the input instance di when constructing di's own demonstration. Section 3.1 defines C as a subset of T, and Section 3.2 says DI converts each labeled input in the training set into a demonstrated input by selecting examples from C. If di ∈ C, then S(di, di) = 1 in Eq. (1), since both the semantic cosine and the feature Jaccard similarity are 1, so di can survive the half-filter and be randomly selected as its own demonstration. In a k-shot setting with k ∈ {5, 10, 20} candidates per feature, this self-selection will occur frequently across the 20 runs. Training would then teach the tagger to copy gold labels directly from the prompt—exactly the behavior that Section 4.5 attributes to ADL—while at test time such copying is impossible because the unseen test input is never in C. This is a fundamental validity threat to the claimed contribution of ADL. Please clarify whether di is removed from C during its own demonstration construction, and if it is not, re-run the experiments with this exclusion and report whether the results change.
  2. [§4.4, Table 2] The feature similarity predictor Mc is trained on the few-shot training set of size k × |F|, but its accuracy is evaluated on pairs drawn from the original full training set. The binary accuracy, ranking accuracy, and Pearson correlation in Table 2 therefore measure the predictor's quality under a data-rich condition that may not match the actual few-shot deployment. If Mc overfits to the small training set or is unreliable under the true distribution, the dual similarity ranking in Eq. (1) degrades to semantic-only selection, undermining the reported gains of ADELL¬DS over DDSS. Please re-evaluate the predictor on the actual few-shot training set or on held-out pairs sampled under the same few-shot regime, and report the corresponding numbers.
  3. [§4.3, Table 1] The claim that 'ADELL achieves the best performance among all methods' is contradicted by the paper's own table. On MIT at k=20, the ablated model ADELL¬ADL achieves 59.93±2.03, which is higher than ADELL's 59.57±2.03. This also contradicts the later statement that both dual similarity and adversarial demonstration learning contribute to the improvement and complement each other. The text should either qualify the claim (e.g., by reporting pairwise wins, significance tests, or average ranks across settings) or provide an explanation, such as known variance, for why the full model underperforms one of its ablations in this condition.
minor comments (4)
  1. [§4.2] The values of the hyperparameters gamma in Eq. (1), alpha and beta in the loss, the ensemble size k in Figure 2(c), the demo pool filtering ratio, and the demo pool size are determined by grid search but are never reported. Please include the final values and search ranges, as well as the demo pool construction details, to make the experiments reproducible.
  2. [§3.2] The symbol k is used both for the number of shots in the few-shot training set and for the number of demonstrated inputs constructed at test time in Figure 2(c). Please use distinct notation or explicitly state whether the ensemble size equals the shot count.
  3. [Abstract and Section 1] There are occasional typographical and formatting issues, such as 'A DELL' with stray spacing in several places and the missing space in 'A DELL ¬DS'. A careful proofread would improve readability.
  4. [§4.4] In the discussion of Table 2, the paper says semantic similarity has 'no correlation' with feature similarity for CoNLL, but the reported Pearson correlation is -0.0519, which is a small negative value rather than exactly zero. Please phrase this as 'negligible' or 'near-zero' to avoid an imprecise claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's method is an empirical pipeline whose components are defined from labels upstream of the reported F1, with no fitted quantity renamed as a prediction.

full rationale

The derivation chain is self-contained against external benchmarks. The dual similarity score in Eq. (1) is a weighted combination of a semantic embedding similarity and a feature Jaccard similarity predictor; the predictor is trained to estimate the Jaccard overlap of entity-type sets (Eq. (2)) from gold annotations, not to predict the final F1. Demonstration selection therefore uses an intermediate label-based score, and the reported test F1 is not an input to any of the component definitions or to the loss in Section 3.3. ADL's label permutation changes training annotation rules, and its effect is measured through independent label-score and attention analyses; no unique-solution claim is imported from a self-citation. The only self-citation (Wu et al. 2020) is a contextual annotation-cost example and carries no derivation weight. Two genuine validity risks exist but are not circularity: (i) the paper says 'A subset C of T is collected as a demo pool' (Sec. 3.1) and 'we apply the demonstration incorporator DI to convert each (labeled) input in the training set into a demonstrated input' (Sec. 3.2), but it never states that the demo pool C excludes the current training input di, so self-demonstration leakage is possible during training; this would be an experimental artifact, not a definitional equivalence, and it cannot affect test-time inputs absent from C; and (ii) the feature predictor evaluation in Section 4.4 draws pairs from the full training set rather than the k-shot set, which may overstate predictor accuracy but does not make the final claim equivalent to its inputs. Since no equation is defined in terms of the result it predicts and no parameter fitted to the reported F1 is re-presented as a prediction, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or linguistic entities. It relies on several trained components and tuned hyperparameters. The free parameters gamma, alpha, beta, and k are not reported, which makes the exact method under-specified. The feature predictor and the NER model both consume the same scarce labels, so the contribution is partly enabled by reusing the few-shot annotations.

free parameters (5)
  • gamma (dual similarity weight) = not reported
    Weight in Eq. (1) balancing feature similarity S_fe and semantic similarity S_se; determined by grid search on validation set, value not given in paper.
  • alpha (main vs adversarial loss weight) = not reported
    Weight in loss combination loss = alpha*lm + (1-alpha)*((1-beta)*le + beta*ll); grid-searched, value not reported.
  • beta (example vs label permutation weight) = not reported
    Controls relative weight of example permutation vs label permutation within adversarial loss; grid-searched, value not reported.
  • k (number of demonstrated inputs at test) = not reported
    At application, DI is applied k times to form k demonstrated inputs for majority voting; k is never specified.
  • demo pool filtering ratio = 0.5
    Half of the ranked examples in each feature subset are removed before random selection, following Gao et al. 2021; treated as a fixed design choice.
assumptions (4)
  • domain assumption Pre-trained BERT/SBERT embeddings provide meaningful semantic similarity for text pairs.
    Used in Eq. (1) via S_se and in the feature predictor's base model; not proven in paper.
  • domain assumption StructShot is a strong base method for few-shot NER and its token embeddings and classifier setup are reliable.
    The model builds on StructShot (Yang and Katiyar 2020) and inherits its assumptions without re-verification.
  • domain assumption A cross-encoder trained on a few hundred pairs can predict feature Jaccard similarity between texts.
    Central to the dual similarity measure; the paper provides limited evidence and a distribution-shifted evaluation.
  • domain assumption Demonstration learning improves low-resource NER.
    Adopted from Gao et al. 2021 and Lee et al. 2022; not re-derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Demonstration Learning for Low-resource NER Using Dual Similarity." pith.science (2026). https://pith.science/paper/UGPRK2F3

@misc{pith2026250715864,
  author       = {Pith},
  title        = {Pith review of: Adversarial Demonstration Learning for Low-resource NER Using Dual Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UGPRK2F3}},
  note         = {Machine review of arXiv:2507.15864}
}
read the original abstract

We study the problem of named entity recognition (NER) based on demonstration learning in low-resource scenarios. We identify two issues in demonstration construction and model training. Firstly, existing methods for selecting demonstration examples primarily rely on semantic similarity; We show that feature similarity can provide significant performance improvement. Secondly, we show that the NER tagger's ability to reference demonstration examples is generally inadequate. We propose a demonstration and training approach that effectively addresses these issues. For the first issue, we propose to select examples by dual similarity, which comprises both semantic similarity and feature similarity. For the second issue, we propose to train an NER model with adversarial demonstration such that the model is forced to refer to the demonstrations when performing the tagging task. We conduct comprehensive experiments in low-resource NER tasks, and the results demonstrate that our method outperforms a range of methods.

Figures

Figures reproduced from arXiv: 2507.15864 by the authors.

Figure 1
Figure 1. An illustrative example of demonstration [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of NER with demonstration. namely, Demonstration Incorporator, Training with Demon￾stration, and Application with Demonstration. In the fig￾ure, we use circles (⃝) and boxes (□) to represent input and demonstrative examples, respectively. Different colors of the boxes indicate different feature types (e.g., orange for [PER]; green for [LOC]). Given an input di , a box (demonstrative example) with a darker … view at source ↗
Figure 3
Figure 3. Similarity prediction. StructShot (see [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: shows the process of creating an adversarial demonstration. Specifically, we first apply example permu￾ [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: Maximum attention score between “Zhang Wei” and each word in the example sentence. The input is: “Zhang Wei went to visit the Eiffel Tower.” The demonstrative example is: “Li Jie aspires to be an outstanding tour guide. Li Jie is [PER].” 5 Conclusion In this paper we i…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    Template-based named entity recognition using BART

    [Cui et al., 2021] Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-based named entity recognition using BART. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021 , pages 1835–1845, Online, August

  2. [5]

    Good examples make a faster learner: Sim- ple demonstration-based learning for low-resource ner

    [Lee et al., 2022] Dong-Ho Lee, Akshen Kadakia, Kang- min Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xi- ang Ren. Good examples make a faster learner: Sim- ple demonstration-based learning for low-resource ner. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics...

  3. [7]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    [Liu et al., 2021a] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602,

  4. [8]

    Gpt understands, too

    [Liu et al., 2021b] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10385,

  5. [9]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    [Liu et al., 2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35,

  6. [11]

    [Ma et al., 2022b] Ruotian Ma, Xin Zhou, Tao Gui, Yid- ing Tan, Linyang Li, Qi Zhang, and Xuanjing Huang

    Association for Computational Lin- guistics. [Ma et al., 2022b] Ruotian Ma, Xin Zhou, Tao Gui, Yid- ing Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. Template-free prompt tuning for few-shot NER. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5721–...

  7. [12]

    [Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych

    Association for Computa- tional Linguistics. [Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages...

  8. [13]

    Introduction to the conll-2003 shared task: Language-independent named entity recognition

    [Sang and De Meulder, 2003] Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Lan- guage Learning at HLT-NAACL 2003 , pages 142–147,

Show all 19 references
  1. [16]

    [Wang et al., 2022] Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng

    Asso- ciation for Computational Linguistics. [Wang et al., 2022] Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Dat...

  2. [17]

    [Wu et al., 2020] Tien-Hsuan Wu, Ben Kao, Anne S

    Association for Computational Linguistics. [Wu et al., 2020] Tien-Hsuan Wu, Ben Kao, Anne S. Y . Che- ung, Michael M. K. Cheung, Chen Wang, Yongxi Chen, Guowen Yuan, and Reynold Cheng. Integrating Domain Knowledge in AI-Assisted Criminal Sentencing of Drug Trafficking Cases, v...

  3. [18]

    Sim- ple and effective few-shot named entity recognition with structured nearest neighbor learning

    [Yang and Katiyar, 2020] Yi Yang and Arzoo Katiyar. Sim- ple and effective few-shot named entity recognition with structured nearest neighbor learning. InProceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6365–6375,

  4. [19]

    Data augmentation for low-resource named entity recognition using backtranslation

    [Yaseen and Langer, 2021] Usama Yaseen and Stefan Langer. Data augmentation for low-resource named entity recognition using backtranslation. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 352–358, 2021

  5. [2003]

    Prototypical networks for few-shot learning

    [Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30,

  6. [2017]

    T-NER: An all-round python library for transformer-based named entity recognition

    [Ushio and Camacho-Collados, 2021] Asahi Ushio and Jose Camacho-Collados. T-NER: An all-round python library for transformer-based named entity recognition. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System...

  7. [2019]

    Making pre-trained language models better few- shot learners

    [Gao et al., 2021] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few- shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P...

  8. [2020]

    Few-shot classification in named entity recognition task

    [Fritzler et al., 2019] Alexander Fritzler, Varvara Lo- gacheva, and Maksim Kretov. Few-shot classification in named entity recognition task. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing , pages 993–1000,

  9. [2021]

    [Dai and Adel, 2020] Xiang Dai and Heike Adel

    Association for Computational Lin- guistics. [Dai and Adel, 2020] Xiang Dai and Heike Adel. An analy- sis of simple data augmentation for named entity recogni- tion. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3861–3867,

  10. [2022]

    A dataset of german legal doc- uments for named entity recognition

    [Leitner et al., 2020] Elena Leitner, Georg Rehm, and Ju- lian Moreno Schneider. A dataset of german legal doc- uments for named entity recognition. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4478–4485,

  11. [2023]

    Label semantics for few shot named entity recognition

    [Ma et al., 2022a] Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. Label semantics for few shot named entity recognition. In Findings of the Association for Computa- tional Linguistics: ACL 2022 , pages 1956–1971, Dubli...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.