Pith. sign in

REVIEW 5 major objections 6 minor 65 references

Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training-time markers give users control and lift long-tail performance.

desk verdict Training-time markers with dual dropout are a real, useful idea, and the long-tail gains hold up through the inferred-marker condition; the fixed-marker results need a baseline-with-template control before the headline numbers are trustworthy. read the letter →

arxiv 2506.14702 v1 pith:P64V5HZC submitted 2025-06-17 cs.CL cs.LG

Daniel D'souza , Julia Kreutzer , Adrien Morisot , Ahmet Üstün , Sara Hooker This is my paper · ORCID
classification cs.CLcs.LG
keywords training-timemarkerslong-tailperformanceinstructiontuningcontrollablegenerationmarkerdropoutmultilingualcoderepairlengthcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the gap between what a model was trained on and what a user asks for at inference time can be closed by attaching a rich set of training-time markers to instruction-tuning data. The authors claim that conditioning generation on a taxonomy of tags, from domain and task to length and quality, improves win rates on open-ended generation and yields disproportionately large gains on rare, long-tail tasks. They also show that marker dropout makes the tags optional, so a model can infer them at inference time or accept them when explicitly provided.

What carries the argument

The central machinery is a taxonomy of training-time markers formatted in a standardized template and placed in both the prompt and the completion, so the model learns associations between these textual attributes and the resulting generation. Each training example carries a list of markers such as <domain>, <task>, <length_bucket>, <language>, <quality_bucket>, and <code_type>, and the training objective conditions the completion on the markers while dropout on the prompt side forces the model to learn to infer missing markers at inference time.

What would settle it

One concrete test would be to compute marker prediction accuracy on a held-out set with human-verified tags, or to re-train the model with markers deliberately corrupted on a subset of the data and check whether the claimed improvements in long-tail performance disappear.

Watch

Extended reading notes

Core claim

The paper's central claim is that adding up to 90 structured training markers across 13 categories to instruction-finetuning data, and letting the model both see and generate these markers, creates controllable generation that performs better on underrepresented use cases. In experiments, the TreasureMarked model improves ArenaHard win rates by 5.7% over a baseline, with 9.1% gains in domains that appear in less than 5% of training data and up to 14.1% relative gains on rare subtasks like CodeRepair. Marker dropout is key: without it the model becomes overly dependent on explicit tags, but with 50% dataset-level dropout it learns to infer markers at inference time, making the tags optional while still gaining control over length, quality, and language.

Load-bearing premise

The load-bearing premise is that the LLM- and dataset-derived markers assigned to the training data are accurate and consistently defined, so that the model learns genuine associations between the tags and the desired output properties.

Editorial extensions

If this is right

  • Users can control generation attributes at inference time by setting or inferring markers, reducing reliance on fragile prompt engineering.
  • Long-tail tasks that appear rarely in the training data stand to gain disproportionately from marker conditioning.
  • Explicit length markers nearly eliminate length-constraint violations (from 36.58% to 1.25% in the fixed setting).
  • Language control improves across 23 languages, with an average absolute gain of 10.98% on a language-confusion benchmark.
  • A second model can annotate an incoming prompt on-the-fly, extending the benefits of markers to unseen user requests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Using an external annotator at inference time to generate markers could serve as a practical deployment pattern, trading one extra model call for improved accuracy on rare use cases.
  • The same taxonomy-based conditioning approach could be applied to other domains, such as specialized reasoning or safety-relevant generation, where rare attributes are worth steering explicitly.
  • The quality and consistency of the LLM-assigned markers is the principal source of uncertainty; if the tags mislabel examples, the model will learn weaker associations.
  • Future work could test whether the benefits persist when markers are assigned by a weaker or cheaper model, or when the taxonomy is expanded beyond the current 23 languages.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes training-time 'treasure markers,' a taxonomy of ninety markers across categories such as length, quality, domain, task, and language, added to instruction-tuning prompts and completions, with dropout to keep them optional at inference. On a proprietary 7B model fine-tuned on 2.7M examples, the authors report that a model trained with markers improves ArenaHard win rate by 5.7% over a no-marker baseline and by 9.1% on domains underrepresented in training data; fixed-marker variants also improve length-constraint following, code repair and translation pass rates, multilingual language control, and machine-translation scores. The paper includes an on-the-fly annotation ablation with Command A and a dropout ablation showing that markers can be inferred at inference time.

Significance. If the reported effects replicate, this is a useful and low-cost training-side intervention: it gives users control levers for quality, length, and language without requiring prompt engineering, and it appears to benefit long-tail tasks disproportionately. The manuscript's strengths include the use of external evaluation benchmarks (Arena-Hard, WMT24++, Language Confusion Benchmark, HumanEvalPack), a machine-translation table with paired significance tests, explicit dropout ablations, and a thorough related-work section. The main caveats are the prompt-enrichment confound in the fixed-marker comparisons, the circular reward-model evaluation of the quality lever, and missing uncertainty quantification for the headline win-rate claims. The general idea of control codes is not new, but the breadth of the taxonomy and its application to general instruction-following are a meaningful step beyond prior single-attribute markers.

major comments (5)
  1. [§4.2.1 (Fig. 5), §4.3 (Table 4), §4.5 (Table 6)] The fixed-marker comparisons lack a marker-only control. In these conditions the TreasureMarked(fixed) test prompts contain a marker template that the Baseline prompts do not contain, so the reported gains (e.g., 14.1% for CodeRepair, 36.58% to 1.25% length violation, +10.98% language pass rate) conflate training-time marker conditioning with inference-time injection of metadata. For Table 4 the two conditions even differ in how the constraint is expressed: the natural-language length instruction is removed and replaced by <length_tokens>, so the comparison is not a clean ablation. Please add a control in which the Baseline model receives the identical marker template at inference, and ideally a shuffled-marker condition, to isolate the effect of the training-time objective.
  2. [§2.1, §3.2] It is not stated whether generated marker lists are stripped before evaluation. Because markers are prepended to completions during training and generation-side dropout is never applied, TreasureMarked models will often emit a <MARKER_LIST> preamble even when the prompt contains no markers. If these preambles are retained in the evaluated outputs, they are an uncontrolled difference from Baseline outputs and could affect both the GPT-4o judge and token-count-based length metrics. Please state explicitly what post-processing is applied to model outputs, and if markers are not stripped, repeat the main evaluations with stripped outputs.
  3. [§4.1.1, Fig. 4] The quality-control result is evaluated with the same reward model that produced the <quality> and <quality_bucket> labels. This circularity means that the 48.21% to 56.5% win-rate curve largely measures the model's ability to mirror the RM's preferences rather than a transferable notion of generation quality. Please re-evaluate the quality lever with an independent judge, such as GPT-4o or human ratings, or with a reward model held out from the labeling process.
  4. [§4.1, Fig. 3, Fig. 4] The headline win-rate numbers are point estimates without error bars or significance tests. Given the finite prompt set and a single judge, the +5.7% overall and +9.1% long-tail differences on Arena-Hard need bootstrap confidence intervals or per-prompt paired tests; the same applies to the quality-lever curve in Fig. 4. The WMT section (Table 5) correctly reports significance tests, so adding equivalent uncertainty quantification to the other evaluations would make the results more comparable and more convincing.
  5. [§3.1] The accuracy of LLM-assigned markers is not validated. The paper states that <domain>, <task>, and <format> are tagged by Command R+ whenever dataset metadata is unavailable, but no annotation agreement, human-validation rate, or consistency check is reported. Since the training objective in Eq. (2) conditions on these labels, and the long-tail code results depend on task labels such as CodeFix and CodeTranslation, noisy or inconsistent labels could undermine the learned associations. Please report annotation quality, for example agreement with a human-annotated subset or a second annotation pass, and preferably include a shuffled-marker control for the main experiments.
minor comments (6)
  1. [Tables 3, 7] The dataset name is misspelled as 'AlpcaEval' in both tables; it should be 'AlpacaEval'.
  2. [Table 5] The caption says 'Bold differences are significant at p≤0.05,' but no bold formatting is visible in the table; please format the significant entries accordingly.
  3. [Table 10] The row labels '0_50', '50_50', and '70_50' are not defined; please state in the caption that the first value is the dataset-level dropout percentage and the second is the sample-level dropout percentage.
  4. [Fig. 2 caption] The caption contains a stray space in 'T reasureMarked'; please fix the typo.
  5. [§4.3] The phrase 'absolute improvements of 35.3%' should be 'a 35.3 percentage-point improvement' because the violation rate changes from 36.58% to 1.25%; the current wording could be read as a relative improvement.
  6. [Eq. (2)] The notation 'dropout(m_d)' is used without a formal definition; please define it as a stochastic masking operator applied independently to each training example.

Circularity Check

1 steps flagged · score 4.0 of 10

Quality-control claim is self-referential (same reward model defines and evaluates the markers); external long-tail benchmarks remain independent, so circularity is partial.

  1. self definitional [Section 4.1.1 (Fixed Treasure Markers), with Table 2 and Appendix A (Quality marker definition)]
    "<quality> Score indicating the quality assigned to a sample as annotated by a human or a Reward-Model (RM). ... As evaluation, we measure the generation quality by the same Reward Model used to score the data during training to compute win rates against the Baseline model."

    The <quality> and <quality_bucket> markers are derived from a proprietary reward model, and the quality-control result is evaluated with that same reward model. Appendix A states 'We utilize a proprietary reward model to assign rewards to a subset of our training data,' and Section 4.1.1 says the win rates are measured 'by the same Reward Model used to score the data during training.' The Figure 4 result therefore shows the model can modulate the exact scoring function that produced the training labels; it is self-consistency with the labeling RM, not independent evidence of quality control.

full rationale

The paper's central long-tail and controllability claims are mostly evaluated against external benchmarks: Arena-Hard uses a GPT-4o judge, HumanEvalPack uses pass@1, WMT uses XCOMET-XL, and the Language Confusion Benchmark uses an external line-level pass rate. These results do not depend on the paper's own marker labels. The one genuine circular step is the quality lever in Section 4.1.1: the same proprietary reward model that defines <quality>/<quality_bucket> is used to measure the win rates, so that result is partially self-referential. I also note a non-circular confound in the fixed-marker settings: TreasureMarked(fixed) appends marker templates at test time (e.g., <length_tokens>199</length_tokens>, <task>CodeFix</task>) while Baseline receives no such template, so headline numbers like CodeRepair +14.1%, length violation 1.25%, and language +10.98% conflate training-time marker conditioning with test-time prompt enrichment. Crucially, the paper also reports TreasureMarked (inferred) results without any test-time marker injection (ArenaHard +5.7%, long-tail domains +9.1%, CodeRepair +6.5%, length violation 36.58% to 24.74%), which provide independent support for the training-time marker claim. No load-bearing self-citation was found; the self-citations to Command R+, Command A, and Aya Expanse are not used to justify the framework's validity. Overall circularity is partial and localized, not global, so the score is 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework relies on the marker taxonomy being meaningful, on automatic annotation being accurate, and on a proprietary reward model's quality scores being valid. These are domain assumptions rather than mathematical axioms. No new physical entities are introduced, and there are no standard-math assumptions beyond ordinary next-token language modeling. Free parameters are hand-chosen thresholds and dropout rates, plus a data-derived quality statistic used in the quality-control experiment.

free parameters (3)
  • length_bucket thresholds = concise: <300 tokens; medium: 300 to 1,000 tokens; long: >1,000 tokens
    Hand-chosen boundaries in Appendix A that define the length markers used in control experiments; no principled or data-driven justification is given.
  • marker dropout rates = dataset-level 50%, sample-level 50%
    Chosen from a small ablation of 0%, 50%, and 70% dataset-level dropout in Section 5.3; 50% is selected as a balance, not by optimization against downstream metrics.
  • quality score percentile = 95th percentile of reward-model scores per language within each quality bucket
    Used to set the <quality> score when combined with <quality_bucket> in Section 4.1.1; derived from training data and tied to the same reward model used in evaluation.
assumptions (4)
  • domain assumption The 90-marker taxonomy and its category definitions capture the output properties users want to control at inference.
    Table 2 and Section 3.1 define the taxonomy; if the categories miss important user needs or misalign with task distributions, the control levers will not transfer.
  • domain assumption Markers assigned by Command R+ and dataset-derived labels are accurate enough for the model to learn correct associations.
    Section 3.1 uses an LLM to tag domain, task, and format when not available; no annotation-quality measurement or human agreement is reported.
  • domain assumption The reward model's quality scores are a valid measure of generation quality.
    Appendix A uses a proprietary reward model to assign <quality> and <quality_bucket> markers, and Section 4.1.1 uses the same reward model to evaluate quality control, making this assumption load-bearing for that result.
  • domain assumption Next-token prediction on the augmented objective p(y,m|dropout(m),x) is sufficient to teach inference-time control.
    Equation (2) defines the training objective; the paper does not prove why this objective transfers to unseen instructions, it is an empirical claim tested only on a proprietary model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers." pith.science (2026). https://pith.science/paper/P64V5HZC

@misc{pith2026250614702,
  author       = {Pith},
  title        = {Pith review of: Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P64V5HZC}},
  note         = {Machine review of arXiv:2506.14702}
}
read the original abstract

One of the most profound challenges of modern machine learning is performing well on the long-tail of rare and underrepresented features. Large general-purpose models are trained for many tasks, but work best on high-frequency use cases. After training, it is hard to adapt a model to perform well on specific use cases underrepresented in the training corpus. Relying on prompt engineering or few-shot examples to maximize the output quality on a particular test case can be frustrating, as models can be highly sensitive to small changes, react in unpredicted ways or rely on a fixed system prompt for maintaining performance. In this work, we ask: "Can we optimize our training protocols to both improve controllability and performance on underrepresented use cases at inference time?" We revisit the divide between training and inference techniques to improve long-tail performance while providing users with a set of control levers the model is trained to be responsive to. We create a detailed taxonomy of data characteristics and task provenance to explicitly control generation attributes and implicitly condition generations at inference time. We fine-tune a base model to infer these markers automatically, which makes them optional at inference time. This principled and flexible approach yields pronounced improvements in performance, especially on examples from the long tail of the training distribution. While we observe an average lift of 5.7% win rates in open-ended generation quality with our markers, we see over 9.1% gains in underrepresented domains. We also observe relative lifts of up to 14.1% on underrepresented tasks like CodeRepair and absolute improvements of 35.3% on length instruction following evaluations.

Figures

Figures reproduced from arXiv: 2506.14702 by the authors.

Figure 1
Figure 1. Tapping into Distributions: (above) illustrates the representation of various length buckets in the training distribution. (below) demonstrates the flexibility of the marker intervention on the mArena Hard test distribution. By modifying the <length_bucket>..</length_bucket> marker, the model can effectively tap into diverse training distributions, even for underrepresented length buckets. data points are not equall… view at source ↗
Figure 2
Figure 2. Modeling data features flexibly with training time markers: Results on the length instruction following on the AlpacaEval-Length-Instructed(LI) dataset. While (a) the baseline violates the length constraint with 36.58%, (b) using the TreasureMarked model and allowing to infer tags on the exact same dataset reduces the violation to 24.7%. (c) Conveying the requirement via an explicitly inserted length marker to the p… view at source ↗
Figure 3
Figure 3. Long tail domains benefit more from training markers: (left) Domain distribution in the training data used to fine-tune the model. (right) Improvements in win rates over the baseline against Gemma2-9B on both majority and minority subsets on Arena-Hard-Auto dataset [Li et al., 2024]. We group the test data into two sets of domains that have high (>5%) and low (< 5%) presence in training data. 3 Taxonomy for training… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Improvement on the Long Tail for Code tasks: (left) Frequency of coding <task>s in the train￾ing dataset. (right) Despite being poorly represented in the training data, CodeRepair achieves a 14.1% relative improvement by leveraging targeted markers during inference fur…
Figure 4
Figure 4. Figure 4: Levers for Controlling Quality: Changing the <quality>, <quality_bucket> mark￾ers at inference time provides control over genera￾tion quality with Win Rates (as measured by inter￾nal Reward Model) going from 48.21% → 56.5% over the Baseline model, demonstrating success…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 20 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Controlling text complexity in neural machine translation

    Sweta Agrawal and Marine Carpuat. Controlling text complexity in neural machine translation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.\ 1549--1564, Hong ...

    doi:10.18653/v1/d19-1166 2019

  3. [3]

    How susceptible are llms to influence in prompts? arXiv preprint arXiv:2408.11865, 2024

    Sotiris Anagnostidis and Jannis Bulian. How susceptible are llms to influence in prompts? arXiv preprint arXiv:2408.11865, 2024

    arXiv 2024

  4. [4]

    i c Barrault. Improving continuous space language models auxiliary features. In Marcello Federico, Sebastian St \

    Walid Aransa, Holger Schwenk, and Lo \"i c Barrault. Improving continuous space language models auxiliary features. In Marcello Federico, Sebastian St \"u ker, and Jan Niehues (eds.), Proceedings of the 12th International Workshop on Spoken Language Translation: Papers, pp.\ 151--158, Da Nang, Vietnam, December 3-4 2015. URL https://aclanthology.org/2015....

    work page 2015

  5. [5]

    Effective domain mixing for neural machine translation

    Denny Britz, Quoc Le, and Reid Pryzant. Effective domain mixing for neural machine translation. In Ond r ej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, and Julia Kreutzer (eds.), Proceedings of the Second Conference on Machine Translation, pp.\ 118--126, Cop...

    work page doi:10.18653/v1/w17-4712 2017

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  7. [7]

    Tagged back-translation

    Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. In Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andr \'e Martins, Christof Monz, Matteo Negri, Aur \'e lie N \'e v \'e ol, Mariana Neves, Matt Post, Marco Turchi, and Karin Vers...

    2019

  8. [8]

    Command a: An enterprise-ready large language model

    Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Rapha \"e l Avalos, et al. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698, 2025

    arXiv 2025

Show all 65 references
  1. [9]

    xcomet: Transparent machine translation evaluation through fine-grained error detection

    Pierre Colombo, Nuno Guerreiro, Ricardo Rei, Daan Van, Luisa Coheur, and Andr \'e Martins. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 2023

    2023

  2. [10]

    Aya expanse: Combining research breakthroughs for a new multilingual frontier

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261, 2024

    2024 arXiv

  3. [11]

    Plug and play language models: A simple approach to controlled text generation

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL https://openr...

    2020

  4. [12]

    Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects

    Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, et al. Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects. arXiv preprint arXiv:2502.12404, 2025

    2025 arXiv

  5. [13]

    Controlling J apanese honorifics in E nglish-to- J apanese neural machine translation

    Weston Feely, Eva Hasler, and Adri \`a de Gispert. Controlling J apanese honorifics in E nglish-to- J apanese neural machine translation. In Toshiaki Nakazawa, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Nobushige Doi, Yusuke Oda, Ond r ej Bojar, Shantipriya Parida, Isao Got...

    2019 doi

  6. [14]

    A natural diet: Towards improving naturalness of machine translation output

    Markus Freitag, David Vilar, David Grangier, Colin Cherry, and George Foster. A natural diet: Towards improving naturalness of machine translation output. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguist...

    2022 doi

  7. [15]

    Are LLM s breaking MT metrics? results of the WMT 24 metrics shared task

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. Are LLM s breaking MT metrics? results of the WMT...

    2024

  8. [16]

    Metadata conditioning accelerates language model pre-training

    Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Metadata conditioning accelerates language model pre-training. arXiv preprint arXiv:2501.01956, 2025

    2025 arXiv

  9. [17]

    spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear, 2017

    2017

  10. [18]

    Document-level information as side constraints for improved neural patent translation

    Laura Jehl and Stefan Riezler. Document-level information as side constraints for improved neural patent translation. In Colin Cherry and Graham Neubig (eds.), Proceedings of the 13th Conference of the Association for Machine Translation in the A mericas (Volume 1: Research Tr...

    2018

  11. [19]

    Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean

    Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. G oogle ' s multilingual neural machine translation system: Enabling zero-shot translation....

    2017 doi

  12. [20]

    Ctrl: A conditional transformer language model for controllable generation

    Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019

    1909 arXiv

  13. [21]

    Domain control for neural machine translation

    Catherine Kobus, Josep Crego, and Jean Senellart. Domain control for neural machine translation. In Ruslan Mitkov and Galia Angelova (eds.), Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017 , pp.\ 372--378, Varna, Bulgaria,...

    2017 doi

  14. [22]

    The stack: 3 tb of permissively licensed source code

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Mu \ n oz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022

    2022 arXiv

  15. [23]

    Navigating the metrics maze: Reconciling score magnitudes and accuracies

    Tom Kocmi, Vil \'e m Zouhar, Christian Federmann, and Matt Post. Navigating the metrics maze: Reconciling score magnitudes and accuracies. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    1999 doi

  16. [24]

    Statistical significance tests for machine translation evaluation

    Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu (eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp.\ 388--395, Barcelona, Spain, July 2004. Association for Computationa...

    2004

  17. [25]

    Gender-aware natural language translation

    James Kuczmarski and Melvin Johnson. Gender-aware natural language translation. Technical Disclosure Commons, 2018. URL https://www.tdcommons.org/dpubs_series/1577

    2018

  18. [26]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  19. [27]

    Like chalk and cheese? on the effects of translationese in MT training

    Samuel Larkin, Michel Simard, and Rebecca Knowles. Like chalk and cheese? on the effects of translationese in MT training. In Kevin Duh and Francisco Guzm \'a n (eds.), Proceedings of Machine Translation Summit XVIII: Research Track, pp.\ 103--113, Virtual, August 2021. Associ...

    2021

  20. [28]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,...

    2021 doi

  21. [29]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  22. [30]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi...

    2021 doi

  23. [31]

    Few-shot learning with multilingual generative language models

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022

  24. [32]

    Cutting down on prompts and parameters: Simple few-shot learning with language models

    Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Associa...

    2022 doi

  25. [33]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  26. [34]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual M...

    2022 doi

  27. [35]

    The vault: A comprehensive multilingual dataset for advancing code understanding and generation

    Dung Nguyen Manh, Nam Le Hai, Anh TV Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi DQ Bui. The vault: A comprehensive multilingual dataset for advancing code understanding and generation. arXiv preprint arXiv:2305.06156, 2023

    2023 arXiv

  28. [36]

    Controlling the reading level of machine translation output

    Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. Controlling the reading level of machine translation output. In Mikel Forcada, Andy Way, Barry Haddow, and Rico Sennrich (eds.), Proceedings of Machine Translation Summit XVII: Research Track, pp.\ 193--203, Dublin...

    2019

  29. [37]

    Understanding and mitigating language confusion in LLM s

    Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Th \'e o Dehaze, and Sebastian Ruder. Understanding and mitigating language confusion in LLM s. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  30. [38]

    Benjamin Marie, Raphael Rubino, and Atsushi Fujita. Tagged back-translation revisited: Why does it really work? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp...

    2020 doi

  31. [39]

    Context dependent recurrent neural network language model

    Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp.\ 234--239, 2012. doi:10.1109/SLT.2012.6424228

    2012

  32. [40]

    Octopack: Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023

    2023 arXiv

  33. [41]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp.\ 248--260. PMLR, 2022

    2022

  34. [42]

    H inted BT : A ugmenting B ack- T ranslation with quality and transliteration hints

    Sahana Ramnath, Melvin Johnson, Abhirut Gupta, and Aravindan Raghuveer. H inted BT : A ugmenting B ack- T ranslation with quality and transliteration hints. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference ...

    2021 doi

  35. [43]

    Aligning large and small language models via chain-of-thought reasoning

    Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain-of-thought reasoning. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long ...

    2024

  36. [44]

    Translationese as a language in ``multilingual'' NMT

    Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. Translationese as a language in ``multilingual'' NMT . In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020 doi

  37. [45]

    Linguistic input features improve neural machine translation

    Rico Sennrich and Barry Haddow. Linguistic input features improve neural machine translation. In Ond r ej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aur \'e lie N \'e v \'e ol, Mariana Neves, ...

    2016

  38. [46]

    Controlling politeness in neural machine translation via side constraints

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Controlling politeness in neural machine translation via side constraints. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computati...

    2016 doi

  39. [47]

    Tag-llm: repurposing general-purpose llms for specialized domains

    Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicol\` o Fusi. Tag-llm: repurposing general-purpose llms for specialized domains. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  40. [48]

    A thorough examination of decoding methods in the era of LLM s

    Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of LLM s. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natura...

    2024 doi

  41. [49]

    Generating diverse translations with sentence codes

    Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. Generating diverse translations with sentence codes. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 1823--1827, Flor...

    2019 doi

  42. [50]

    Aya dataset: An open-access collection for multilingual instruction tuning

    Shivalika Singh, Freddie Vargus, Daniel Dsouza, B \"o rje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.0...

    2024 arXiv

  43. [51]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i...

    2025

  44. [52]

    Multi-domain adaptation in neural machine translation through multidimensional tagging

    Emmanouil Stergiadis, Satendra Kumar, Fedor Kovalev, and Pavel Levin. Multi-domain adaptation in neural machine translation through multidimensional tagging. In Janice Campbell, Ben Huyck, Stephen Larocca, Jay Marciano, Konstantin Savenkov, and Alex Yanishevsky (eds.), Proceed...

    2021

  45. [53]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  46. [54]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st A...

    2023 doi

  47. [55]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in N...

    2022

  48. [56]

    Prompt engineering 101 prompt engineering guidelines from a linguistic perspective

    Han Wenjuan, Wei Xiang, Cui Xingyu, Cheng Ning, Jiang Guangyuan, Qian Weinan, and Zhang Chi. Prompt engineering 101 prompt engineering guidelines from a linguistic perspective. In Maosong Sun, Jiye Liang, Xianpei Han, Zhiyuan Liu, and Yulan He (eds.), Proceedings of the 23rd C...

    2024

  49. [57]

    Cross-lingual few-shot learning on unseen languages

    Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. Cross-lingual few-shot learning on unseen languages. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the...

    2022

  50. [58]

    Language tags matter for zero-shot neural machine translation

    Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. Language tags matter for zero-shot neural machine translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.\ 3001--3007, Onl...

    2021 doi

  51. [59]

    Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings

    Yangjian Wu and Gang Hu. Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp.\ ...

    2023 doi

  52. [60]

    Controlling the voice of a sentence in J apanese-to- E nglish neural machine translation

    Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. Controlling the voice of a sentence in J apanese-to- E nglish neural machine translation. In Toshiaki Nakazawa, Hideya Mino, Chenchen Ding, Isao Goto, Graham Neubig, Sadao Kurohashi, Ir. Hammam Riza, and Pus...

    2016

  53. [61]

    Tailor: A soft-prompt-based approach to attribute-based controlled text generation

    Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. Tailor: A soft-prompt-based approach to attribute-based controlled text generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeti...

    2023 doi

  54. [62]

    Exploring the effectiveness of prompt engineering for legal reasoning tasks

    Fangyi Yu, Lee Quartey, and Frank Schilder. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 13582--13596, Toron...

    2023 doi

  55. [63]

    Following length constraints in instructions

    Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Following length constraints in instructions. arXiv preprint arXiv:2406.17744, 2024

    2024 arXiv

  56. [64]

    A survey of controllable text generation using transformer-based pre-trained language models

    Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-based pre-trained language models. ACM Comput. Surv., 56 0 (3), October 2023. ISSN 0360-0300. doi:10.1145/3617680. URL https://doi.org/10.1145/3617680

    2023 doi

  57. [65]

    Seq2sql: Generating structured queries from natural language using reinforcement learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017

    2017 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.