REVIEW 4 cited by
A Contrastive Framework for Neural Text Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.
Forward citations
Cited by 4 Pith papers
-
Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
LLM probability estimates violate the law of total probability across partitions, and subgroup-aggregated estimates often beat direct population-level estimates (the macro fallacy).
-
Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
Generating several candidate translations per source sentence for knowledge distillation yields better small multilingual translators than standard single-hypothesis distillation, especially in low-resource settings.
-
DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMs
DecoRTL combines token-class-aware temperature adjustment with contrastive top-K reranking to improve synthesizability and functional correctness of LLM-generated Verilog.
-
From Image Captioning to Visual Storytelling
Visual storytelling improves by treating it as image captioning followed by language-to-language story generation, with a new 'ideality' metric to gauge distance from an oracle.
Discussion (0). Continue with ORCID to comment.