Pith. sign in

REVIEW 3 major objections 5 minor 49 references

RoToR: Towards More Reliable Responses for Order-Invariant Inputs

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen LLM can be made order-invariant in a single forward pass by assigning position IDs from a rotated global ordering of the input segments.

desk verdict A practical, training-free method for reducing positional bias that mostly delivers on LitM and KGQA, but the paper overstates both the invariance guarantee and the statistical evidence. read the letter →

arxiv 2502.08662 v3 pith:I5MAQ3YU submitted 2025-02-10 cs.CL cs.AI

Soyoung Yoon , Dongha Ahn , Youngwon Lee , Minkyu Jung , HyungJoo Jang , Seung-won Hwang This is my paper · ORCID
classification cs.CLcs.AI
keywords positionalbiasorder-invariantinputslistwiserotationalpositionIDassignmentlostinthemiddlezero-shotorderinvarianceselectiveroutinglanguagemodelreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models are biased by the order of list-like inputs: shuffling retrieved documents, table rows, or multiple-choice options can change answers. This paper claims that a frozen causal LLM can be made genuinely order-invariant with a single global sort of the input segments followed by circular position-ID assignment, so that each segment takes a turn appearing last when it acts as the query. The method, RoToR, is designed to avoid the train–test distribution mismatch and the per-query attention sorting of earlier zero-shot invariant models. Because real tasks mix order-invariant inputs with order-sensitive ones, the paper adds Selective Routing, which trusts whichever of the invariant or original model is more confident. If the claim is right, reliable answers on listwise inputs no longer require running many permutations or retraining.

What carries the argument

The load-bearing mechanism is circular position-ID assignment over a single global segment ordering. Given a deterministic global order of the $k$ input segments, RoToR forms the same directed cycle for every query: starting from the query segment, the remaining segments follow in global order, and the query segment gets the largest position IDs so that it can attend to all others. Prefix and suffix tokens keep standard causal positions, and all suffix tokens see one shared rotation, so the added cost is one global sort, $O(k \log k)$, with no extra attention computation. This replaces PINE's per-query, per-head, per-layer attention-score sorting, which the paper identifies as the source of distribution mismatch, numerical instability, and ordering collisions.

What would settle it

Take an order-invariant task with $k$ distinct input segments, enumerate all $k!$ orderings, and check whether RoToR's greedy-decoded answers are identical across every ordering; any statistically reliable difference would refute the claimed invariance. A cheaper falsifier is to compare RoToR's hidden activations for two permutations of the same segment set at corresponding rotated positions—if they differ for any token, the mechanism is not exactly permutation-invariant.

Watch

Extended reading notes

Core claim

The central claim is that order-invariance for decoder-only LMs is best achieved not by eliminating cross-segment attention or by re-sorting key segments for every query according to attention scores, but by choosing one deterministic global ordering of the segments and rotating it circularly when assigning position IDs. Under RoToR, the segment acting as query is placed at the end of the segment sequence, the other segments keep their relative global order, and suffix and generated tokens see a single fixed arrangement. The paper argues this keeps the model close to its pretrained positional distribution while still letting every segment attend to all others, and it reports that any global sorting strategy with this circular assignment outperforms the attention-based PINE scheme. On Lost in the Middle and knowledge-graph QA, RoToR improves accuracy and reduces variance across shuffles; on MMLU, RoToR alone is competitive but not always better, and adding Selective Routing makes the combination match or beat the original model across all answer orderings.

Load-bearing premise

The method assumes that rotating a single global ordering of position IDs is a small enough distributional change that a frozen pretrained LM keeps behaving sensibly, and that this rotation by itself makes the model truly order-invariant; on top of that, Selective Routing assumes that the maximum output-token probability is a reliable confidence signal for choosing between the two models.

Editorial extensions

If this is right

  • A frozen, instruction-tuned LLM can be made order-invariant without any training, using one forward pass instead of evaluating multiple permutations.
  • Order-invariance no longer requires computing attention scores for sorting; the global sort is done once, making the method cheaper as the number of segments grows.
  • Collisions from tied attention scores are largely eliminated: lexical sorting only ties when two segment texts are literally identical.
  • Selective Routing lets a single deployment handle mixed listwise inputs, such as multiple-choice questions containing 'None of the above' options, without sacrificing the original model's accuracy.
  • The same mechanism extends beyond short lists: experiments with roughly 10k-token multi-hop QA contexts show RoToR remains effective and stable under severe order perturbations, where PINE runs out of memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the paper does not pursue: if order-invariance is exact, RoToR's per-instance outputs should be identical across all permutations, so per-instance agreement across shuffles is a sharper test than average accuracy.
  • The global ordering is a free design choice, so one could sort by task-relevant keys (e.g., a numerical priority or semantic cluster) to combine order-invariance with a desired reading order—an untested extension.
  • Selective Routing's confidence comparison could serve as an order-sensitivity detector more generally: when the original and invariant models disagree, the input is likely order-sensitive or ambiguous, which may be useful for uncertainty estimation.
  • The circular-assignment idea is stated in terms of absolute position IDs, and an untested question is whether it transfers to relative positional encodings or to bidirectional models, where the same rotation could be implemented as a permutation of the key-value order.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes RoToR, a zero-shot method for making decoder-only LMs more order-invariant on listwise inputs. RoToR globally sorts input segments (e.g., lexically) and assigns position IDs via a circular arrangement, so that each segment can appear 'last' from its own perspective without per-query attention-based reordering. The paper also proposes Selective Routing, which routes between the original model and the invariant model based on maximum token probability plus a tuned bias term, aiming to handle benchmarks that mix order-invariant and order-sensitive inputs. Experiments cover Lost in the Middle (LitM), Mintaka KGQA, and MMLU, with baselines including PCW, Set-Based Prompting, PINE, and the original model, plus analyses of efficiency, perplexity, collision rate, long-context inputs, and template robustness.

Significance. If the central claim held, the contribution would be practical: RoToR is training-free, zero-shot, and avoids PINE's O(n^2 d) attention-computation overhead, using only a single global sort with O(k log k) cost; the efficiency comparison in Table 4 is detailed and useful. The paper also has several strengths: the deterministic global-sort mechanism has no fitted parameters; the only tuned quantity (alpha=0.2) is disclosed and selected on a validation split; the Oracle upper bound is explicitly labeled; and the code is released. However, the exact-invariance claim is contradicted by the paper's own shuffle-variance numbers, and the statistical support for robustness is methodologically weak. The empirical results are mostly favorable, but the current version overstates what is established, so the claims need to be revised or the experiments need to be strengthened.

major comments (3)
  1. [§3.2 and Tables 2, 7] The claim that RoToR 'guarantee[s] invariance with a single forward pass' (§3.2; also §2.2) is not supported by the reported data. Table 2 gives nonzero seed-to-seed standard deviations for RoToR (e.g., Llama-3.1-8B-Instruct, N=30: 0.05/0.05/0.02; Qwen1.5-7B-Chat, N=50: 0.06/0.05/0.04), and Table 7 lists before/after-shuffle differences for RoToR, including 2.8 points on MMLU with Selective Routing. If the global sort plus circular position assignment canonicalized the input, all shuffled runs would be identical and these numbers would be zero. Since the method as described reassigns position IDs while tokens retain their original physical order (Fig. 2 caption), the causal mask still depends on the input permutation, so invariance is at best approximate. Please either physically reorder segments to make the output exactly permutation-invariant, or revise the invariance claim and the related significance statements to describe approximate invariance.
  2. [Appendix I; Sec. 5] The robustness argument in Sec. 5 relies on a paired t-test that pools nine heterogeneous observations (Mintaka at two ndoc values for three backbones and MMLU for three backbones), uses seed-averaged scores as single points, and ignores within-condition variance. For RoToR the test gives t=2.23, df=8, p≈0.056, which is non-significant at α=0.05, but a non-significant result in this pooled design is not evidence of order-invariance; moreover Table 7 shows RoToR+SR dropping 2.8 points on MMLU, the same as the original model. Report per-condition confidence intervals, effect sizes, or a model that accounts for the repeated-measures structure, and do not claim statistical evidence of robustness from this table.
  3. [Table 1 and Sec. 5] The caption of Table 1 states that RoToR 'shows the best performance across all setups,' and Sec. 5 concludes that 'any type of global sorting, with the use of circular assignment is superior than PINE.' Both statements are contradicted by Table 1 for Qwen1.5-4B-Chat with ndoc=30: e.g., at gold index 0, PINE scores 46.4 while RoToR-lexical scores 44.6, and PINE is also ahead at most other gold positions for that configuration. Please qualify the claim to the specific sorting variants and model sizes where it actually holds.
minor comments (5)
  1. [Appendix I] The heading 'Statisticial significance' contains a typo; it should be 'Statistical significance.'
  2. [Throughout] The method name is inconsistent: the title and abstract use 'RoToR' while many tables and appendix headings use 'ROTOR.' Please standardize the notation.
  3. [Appendix F] The sentence 'We Except for the experiments on Llama-3.1-70B-Instruct, we only use a single A6000 GPU for all of the experiments' is ungrammatical and should be rewritten.
  4. [Appendix J] The word 'consistantly' is a typo for 'consistently' in the sentence about LongBench-2WikiMultihopQA.
  5. [Table 4] The panel (a) label 'Overhead FLOPs, relative to original model' with a 'Reduction' column is ambiguous: clarify whether the reduction is relative to the PINE overhead or to the original model's FLOPs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RoToR is a deterministic, parameter-free position-ID transformation, and the only tuned hyperparameter (alpha in Selective Routing) is selected on a validation split and applied to a held-out test set.

full rationale

The central derivation is self-contained rather than circular. RoToR (Sec. 3.2) is a deterministic position-ID transformation: segments are sorted by a global key (lexical, MonoT5, or frequency) and assigned circular position IDs; no parameter is fitted to the evaluated benchmarks. The claimed invariance is a property of the reordering/ID-assignment mechanism, not an output derived from the evaluation data. The only fitted quantity, alpha=0.2 in Selective Routing (Sec. 3.3), is explicitly tuned on the MMLU validation split (Appendix E, Table 6) and then applied to the 14,015-question test set, which is standard hyperparameter selection rather than a fitted input being relabeled as a prediction. The RoToR+SR Oracle row is explicitly labeled as an upper bound ('perfect routing accuracy', Table 3), not presented as a realized prediction. Citations to the authors' own prior work (Lee et al. 2025a,b; Hwang and Chang 2007) appear only in the related-work survey and are not load-bearing in the method's derivation. Concerns about the paired t-test pooling heterogeneous settings, or about nonzero shuffle standard deviations contradicting the claimed exact invariance, are correctness/validity criticisms rather than circularity: they do not show that any reported result reduces to its input by construction.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free-parameter footprint is small: one routing bias alpha fitted per dataset. The main assumptions are domain assumptions about the mechanism of positional bias in pretrained LMs and the reliability of confidence scores for routing.

free parameters (1)
  • Selective Routing bias alpha = 0.2
    Tuned on the MMLU validation set (Appendix E) to favor the original model; applied to the test set. It is a hyperparameter of the routing method, not a scientific constant.
assumptions (2)
  • domain assumption A causal decoder-only LM's positional bias is primarily caused by positional encoding and causal attention mask, and can be mitigated by reassigning position IDs without retraining.
    Invoked throughout Sec. 3; the entire method assumes frozen LMs can be made order-invariant by position-ID manipulation alone.
  • domain assumption The maximum output token probability is a reliable proxy for the correctness of a model's answer for routing decisions.
    Used in Selective Routing (Sec. 3.3); the paper builds on Wei et al. (2024) for the correlation between confidence and order sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RoToR: Towards More Reliable Responses for Order-Invariant Inputs." pith.science (2026). https://pith.science/paper/I5MAQ3YU

@misc{pith2026250208662,
  author       = {Pith},
  title        = {Pith review of: RoToR: Towards More Reliable Responses for Order-Invariant Inputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5MAQ3YU}},
  note         = {Machine review of arXiv:2502.08662}
}
read the original abstract

Mitigating positional bias of language models (LMs) for listwise inputs is a well-known and important problem (e.g., lost-in-the-middle). While zero-shot order-invariant LMs have been proposed to solve this issue, their success on practical listwise problems has been limited. In this work, as a first contribution, we identify and overcome two limitations to make zero-shot invariant LMs more practical: (1) training and inference distribution mismatch arising from modifying positional ID assignments to enforce invariance, and (2) failure to adapt to mixture of order-invariant and sensitive inputs in practical listwise problems. Then, to overcome these issues we propose (1) RoToR, a zero-shot invariant LM for genuinely order-invariant inputs with minimal modifications of positional IDs, and (2) Selective Routing, an adaptive framework that handles both order-invariant and order-sensitive inputs in listwise tasks. On the Lost in the middle (LitM), Knowledge Graph QA (KGQA), and MMLU benchmarks, we show that RoToR with Selective Routing can effectively handle practical listwise input tasks in a zero-shot manner (https://github.com/soyoung97/RoToR)

Figures

Figures reproduced from arXiv: 2502.08662 by the authors.

Figure 1
Figure 1. Self-attention alteration from order-invariant [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Attention mask and positional ID modifications for segment-wise order invariance using example input [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparing the ordering of 5 segments (S1 - S5) of PINE ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Illustration of Selective Routing (Sec. E). further applicability. Our design is partly based on the finding from Wei et al. (2024) that there is correlation between task difficulty (which is in turn correlated with confidence values) and the model’s sensitivity to ord…
Figure 5
Figure 5. Figure 5: Results on the Lost-in-the-middle benchmark. Visualization of the [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Illustration of ordering 7 rows by 2 different global sort options, (1) lexical sort based on token ids, or (2) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Example input for the lost in the middle dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Example input for the lost in the middle dataset, without indexing by numbers. Prompt for the Llama-3.1- [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Example input for the lost in the middle dataset, without indexing by numbers, prompt for the Qwen1.5- [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Example input for the Mintaka dataset. MMLU Prefix: The following are multiple choice questions (with answers) about moral disputes. Norcross agrees that if a being is incapable of moral reasoning, at even the most basic level, then it cannot be Parallel texts: A. a b…
Figure 11
Figure 11. Figure 11: Example input for the MMLU benchmark. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 15 canonical work pages

  1. [1]

    Meta AI. 2024. https://llama.meta.com/llama3 Build the future of ai with meta llama 3

    work page 2024

  2. [2]

    Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. https://arxiv.org/abs/2402.01781 When benchmarks are targets: Revealing the sensitivity of large language model leaderboards . Preprint, arXiv:...

    arXiv 2024

  3. [3]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. https://arxiv.org/abs/2307.11088 L-eval: Instituting standardized evaluation for long context language models . Preprint, arXiv:2307.11088

    arXiv 2023

  4. [4]

    Park, and Sung Ju Hwang

    Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C. Park, and Sung Ju Hwang. 2023. https://arxiv.org/abs/2310.12836 Knowledge-augmented language model verification . Preprint, arXiv:2310.12836

    arXiv 2023

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . Preprint, arXiv:2308.14508

    arXiv 2024

  6. [6]

    Lee, and Mengdi Wang

    Tianle Cai, Kaixuan Huang, Jason D. Lee, and Mengdi Wang. 2023. https://openreview.net/forum?id=jH580PKkPw Scaling in-context demonstrations with structured attention . In Workshop on Efficient Systems for Foundation Models @ ICML2023

    work page 2023

  7. [7]

    Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939

    arXiv 2024

  8. [8]

    Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2023. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455

    arXiv 2023

Show all 49 references
  1. [9]

    Anshuman Chhabra, Hadi Askari, and Prasant Mohapatra. 2024. https://arxiv.org/abs/2401.01989 Revisiting zero-shot abstractive summarization in the era of large language models from the perspective of position bias . Preprint, arXiv:2401.01989

    2024 arXiv

  2. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  3. [11]

    Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. https://arxiv.org/abs/2406.19470 Changing answer order can decrease mmlu accuracy . Preprint, arXiv:2406.19470

    2024 arXiv

  4. [12]

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022 a . Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022

    2022

  5. [13]

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022 b . https://arxiv.org/abs/2203.16634 Transformer language models without positional encodings still learn positional information . Preprint, arXiv:2203.16634

    2022 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  7. [15]

    Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:...

    2024 arXiv

  8. [16]

    Seung-won Hwang and Kevin Chen-chuan Chang. 2007. https://doi.org/10.1145/1206049.1206054 Optimizing top-k queries for middleware access: A unified cost-based approach . ACM Trans. Database Syst., 32(1):5–es

    2007

  9. [17]

    He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qianguo, Zhang Songxin, Xie Zejian, et al. 2023. Never lost in the middle: Improving large language models via attention strengthening question answering. arXiv preprint arXiv:2311.09198

    2023 arXiv

  10. [18]

    Kosiorek, Seungjin Choi, and Yee Whye Teh

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In ICML

    2019

  11. [19]

    Youngwon Lee, Seung won Hwang, Daniel Campos, Filip Graliński, Zhewei Yao, and Yuxiong He. 2025 a . Cord: Balancing consistency and rank distillation for robust retrieval-augmented generation. NAACL

    2025

  12. [20]

    Youngwon Lee, Seung won Hwang, Daniel Campos, Filip Graliński, Zhewei Yao, and Yuxiong He. 2025 b . Inference scaling for bridging retrieval and augmented generation. NAACL

    2025

  13. [21]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939

    2023 arXiv

  14. [22]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  15. [23]

    Reid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang, and Cynthia Dwork. 2024. https://arxiv.org/abs/2406.06581 Set-based prompting: Provably solving the language model order dependency problem . Preprint, arXiv:2406.06581

    2024 arXiv

  16. [24]

    Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro

    Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. 2019 a . https://arxiv.org/abs/1811.01900 Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs . Preprint, arXiv:1811.01900

    2019 arXiv

  17. [25]

    Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro

    Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. 2019 b . https://arxiv.org/abs/1903.02541 Relational pooling for graph representations . Preprint, arXiv:1903.02541

    2019 arXiv

  18. [26]

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. https://arxiv.org/abs/2003.06713 Document ranking with a pretrained sequence-to-sequence model . Preprint, arXiv:2003.06713

    2020 arXiv

  19. [27]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  20. [28]

    Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427

    2023 arXiv

  21. [29]

    Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483

    2023 arXiv

  22. [30]

    Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. https://doi.org/10.18653/v1/2023.acl-long.352 Parallel context windows for large language models . In Proceedings of the 61st Ann...

    2023 doi

  23. [31]

    Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. https://aclanthology.org/2022.coling-1.138/ Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering . In Proceedings of the 29th International Conference on Computational Linguistics, pages ...

    2022

  24. [32]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297

    2020 arXiv

  25. [33]

    Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2024. https://arxiv.org/abs/2310.07712 Found in the middle: Permutation self-consistency improves listwise ranking in large language models . Preprint, arXiv:2310.07712

    2024 arXiv

  26. [34]

    Kakade, Hao Peng, and Heng Ji

    Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. 2024. https://arxiv.org/abs/2407.01100 Eliminating position bias of language models: A mechanistic approach . Preprint, arXiv:2407.01100

    2024 arXiv

  27. [35]

    Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. https://arxiv.org/abs/2406.03009 Unveiling selection biases: Exploring order and token sensitivity in large language models . Preprint, arXiv:2406.03009

    2024 arXiv

  28. [36]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025

    2023 arXiv

  29. [37]

    Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. https://arxiv.org/abs/2203.00274 Tableformer: Robust transformer modeling for table-text encoding . Preprint, arXiv:2203.00274

    2022 arXiv

  30. [38]

    Kejuan Yang, Xiao Liu, Kaiwen Men, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2023. https://arxiv.org/abs/2305.15262 Revisiting parallel context windows: A frustratingly simple alternative and chain-of-thought deterioration . Preprint, arXiv:2305.15262

    2023 arXiv

  31. [39]

    Howard Yen, Tianyu Gao, and Danqi Chen. 2024. https://arxiv.org/abs/2402.16617 Long-context language modeling with parallel context encoding . Preprint, arXiv:2402.16617

    2024 arXiv

  32. [40]

    Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. 2024. Mitigate position bias in large language models via scaling a single dimension. arXiv preprint arXiv:2406.02536

    2024 arXiv

  33. [41]

    Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024 a . https://arxiv.org/abs/2312.09542 Marathon: A race through the realm of long context with large language models . Preprint, arXiv:2312.09542

    2024 arXiv

  34. [42]

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024 b . Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797

    2024 arXiv

  35. [43]

    Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023. https://arxiv.org/abs/2306.14321 Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations . Preprint, arXiv:2306.14321

    2023 arXiv

  36. [44]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024 a . https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024 b . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  38. [46]

    Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. 2024. https://arxiv.org/abs/2404.18585 Freb-tqa: A fine-grained robustness evaluation benchmark for table question answering . Preprint, arXiv:2404.18585

    2024 arXiv

  39. [47]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631

    2023 arXiv

  40. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.