Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

CodeContests+: High-Quality Test Case Generation for Competitive Programming

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Generator-Validator pair of LLM agents rewrites the test cases of a competitive-programming dataset, and the new tests separate correct from incorrect submissions far more accurately than the original ones.

desk verdict Useful dataset and agent system, but TPR/TNR evidence conflates new test inputs with new checkers; the central quality claim needs an ablation to hold up. read the letter →

arxiv 2506.05817 v1 pith:GCZAP76H submitted 2025-06-06 cs.SE cs.CL

Zihan Wang , Siyao Liu , Yang Sun , Hongyan Li , Kai Shen This is my paper
classification cs.SEcs.CL
keywords testcasegenerationcompetitiveprogrammingLLMagentsGenerator-ValidatorsystemCodeContests+reinforcementlearningTPRandTNRevaluationdatasetquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Competitive programming datasets have abundant problem statements and solutions but rarely release official test cases, so dataset builders must generate their own; the paper's claim is that the quality of those generated test cases, not just their quantity, controls both evaluation accuracy and reinforcement-learning signal. The paper proposes a Generator-Validator agent system in which one LLM writes an executable test generator and a second LLM writes an executable validator that checks every stated input constraint and feeds violations back for revision. Replacing the mutation-based test cases of CodeContests with G-V-produced cases yields CodeContests+, which the paper reports scores much higher true positive and true negative rates on 1.72 million real submissions. If correct, this makes large-scale, trustworthy code-training datasets feasible from public problem data and makes test-case quality a measurable, filterable property of a dataset.

What carries the argument

The paper's central mechanism is the Generator-Validator agent system. The Generator agent produces an executable generator program, parameterized by command-line options such as data size and case type, that can emit arbitrarily many test inputs; the Validator agent produces an executable validator program that checks every stated input constraint, including structural ones, and returns precise errors to the Generator for revision. A seeded random library makes the generated test sets reproducible across platforms, and a Checker agent supplies custom judging logic for problems with multiple valid outputs, such as topological sorts or floating-point answers.

What would settle it

A concrete check would be to take a random sample of released CodeContests+ test inputs, have independent validators written from scratch by humans or by a separate LLM instance for the same problem statements, and count how many inputs violate any stated constraint; if the violation rate is non-negligible, the 'verified' label and the associated TPR/TNR advantage would be called into question.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM-based Generator-Validator agent system can produce test cases for competitive programming problems that are both more correct and more comprehensive than those in existing large datasets. Applying it to CodeContests yields CodeContests+, in which every one of the released test inputs passed a validator program written to enforce the problem statement's constraints, including structural constraints such as acyclicity. Measured on 1.72 million real contestant submissions labeled correct or incorrect, per-problem TPR (fraction of correct solutions that pass) and TNR (fraction of incorrect solutions that fail) are substantially higher for CodeContests+ than for CodeContests, with the largest gain in TPR. The paper further claims that training a 32B reasoning model with reinforcement learning on the high-quality subset gives consistent performance gains over training on the original dataset.

Load-bearing premise

The claim that CodeContests+ is a verified dataset depends on the LLM-written validators catching every constraint in each problem statement; the paper concedes that a validator can overlook constraints, so a small number of invalid inputs can still pass.

Editorial extensions

If this is right

  • At equal TPR and TNR thresholds, CodeContests+ 5x contains nearly twice as many qualified problems as CodeContests; even the 1x version, with about a quarter of the tests, yields over 80% more qualified problems.
  • Reinforcement learning on the high-quality subset produces consistent pass@1 gains across easy, medium, and hard benchmark levels compared with training on the original dataset.
  • Because test inputs are produced by executable generators, users can generate as many reproducible tests as the compute budget allows, not just the pre-released versions.
  • Per-problem TPR and TNR scores make test-case quality a filterable dataset property, letting users trade evaluation cost against accuracy by choosing a threshold.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same G-V pipeline were run on the other publicly available programming problems the paper estimates exist, test-case quality screening could be standardized before any RL run, but the paper does not test this.
  • The overlooked-constraint failure mode implies that 'verified' should be read as 'validated against the constraints the LLM noticed'; formalizing constraints or adding a second independent validator would be a natural hardening step the authors leave to future work.
  • The checker agent's richer error messages are not used in the current RL reward, which is binary; using them as dense feedback is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an LLM-based Generator-Validator (G-V) agent system to construct test inputs for competitive programming problems, applies it to CodeContests to produce CodeContests+, and claims that the new dataset has 'verified' test cases of significantly higher quality than the original. Quality is measured in two ways: (i) TPR/TNR of the two test suites against 1.72 million labeled contestant submissions (Section 4.4, Fig. 3), and (ii) an RL training comparison in which a 32B model trained on a filtered subset CodeContests+HQ outperforms one trained on unfiltered CodeContests on LiveCodeBench (Section 4.5, Table 3). The paper also contributes custom checker programs for multi-solution problems and releases the dataset, generator/validator programs, and a cloud evaluation service.

Significance. If the central claim holds, CodeContests+ would be a valuable resource for both evaluation and RL training in competitive programming: it would provide larger and more reliable test suites, per-problem quality metadata, and a scalable pipeline for constructing test cases for future datasets. The paper's engineering effort is substantial: the evaluation involves more than 300 million program executions, the data release includes five pre-generated test-set versions plus generators and validators, and the custom checker infrastructure addresses a real gap in existing datasets. These are genuine contributions. However, the headline quantitative claims currently rest on a comparison that changes two variables at once, and on a ground-truth labeling assumption that the paper itself shows to be unreliable in Appendix D.3; the significance of the dataset is therefore not yet established at the level claimed.

major comments (4)
  1. [Section 4.4, Fig. 3] The TPR/TNR comparison does not isolate test-input quality: the intervention simultaneously replaces the test inputs and adds custom checkers to all problems (Section 4.3). The paper itself attributes a large part of CodeContests' near-zero TPR to 'CodeContests does not provide custom checkers for multi-solution problems' (Section 4.4, Results). Consequently, the observed TPR improvement could largely reflect the checker accepting multiple valid outputs rather than the generated inputs being better. To support the load-bearing claim of 'significantly higher test case quality', the authors should run the original CodeContests test inputs through the same custom checker infrastructure, or restrict the comparison to problems with unique outputs where checker effects are absent, and report whether the advantage persists.
  2. [Section 4.5, Table 3] The RL experiment compares CodeContests+HQ, a subset filtered by checker-inclusive TPR&TNR>=0.9, against the unfiltered CodeContests set. This confounds test-case quality with dataset filtering: any quality-filtered subset could yield better RL training signal than an unfiltered one, regardless of whether the new test cases are better than the old ones. A controlled comparison should use the same filtering criterion for CodeContests (i.e., select CodeContests problems with TPR&TNR>=0.9 under the original evaluation) or otherwise match the problem sets, and ideally report multiple RL runs or seed variance, since Table 3 shows a single run without any variance estimate.
  3. [Appendix D.3 and Section 4.4] The TPR/TNR analysis treats official contestant labels as ground truth, but Appendix D.3 demonstrates that a submission can pass the official tests and be labeled correct while being actually incorrect (the program outputs 2 when the correct answer is 3). This means a TPR decrease in CodeContests+ can reflect correctly rejecting a mislabeled positive, not a flaw in the test case; conversely, CodeContests' high TNR may be inflated by mislabeled negatives. The paper's interpretation of TPR as measuring 'the test case's correctness' (Section 4.4) is therefore not clean. The authors should quantify label noise (e.g., by sampling submissions with conflicting verdicts between the two test suites and manually auditing them) or temper the claim.
  4. [Section 3.2 and Contribution 2] The abstract and contribution list describe CodeContests+ as having 'verified' test cases, yet Section 3.2 explicitly concedes that the second type of validator error (overlooked constraints) 'can still result in a small number of incorrect data being generated'. Since no automatic supervision exists for that error class, the label 'verified' overstates the guarantee. This does not invalidate the dataset, but the paper should qualify the claim (e.g., 'validator-checked' rather than 'verified') and report the estimated residual error rate from the case studies in Section 4.4.
minor comments (6)
  1. [Abstract] The abstract says 'CodeContestsPlus' instead of 'CodeContests+' in the phrase 'the quality of test cases in CodeContestsPlus'.
  2. [Section 4.2, Table 2] The entry '25/44/62/80/98/∞' is unclear: the infinity symbol presumably denotes the dynamic generation mode, but this should be stated explicitly in the table caption.
  3. [Section 4.4] The text says 'we sampled 100 positive samples and 100 negative samples for each problem' but then says 'we selected 10,166 problems that contained at least 10 positive and negative samples'; the discrepancy between 100 and 10 should be reconciled.
  4. [Section 4.5] The paper defines avg@15 as 'the average of pass@1 from 15 independent responses' but reports the metric in Table 3 as Pass@1; please make the metric name consistent.
  5. [Appendix D.2] The word 'commnads' is a typo for 'commands' in the generator command listing.
  6. [Appendix C] The notation '4c16g' and '2c4g' is not defined; please state that these are CPU/memory specs of the judging and execution pods.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the quality claims are empirical measurements on external submissions and an external benchmark, not derivations from fitted parameters.

full rationale

This paper is an empirical dataset-construction and evaluation paper, not a derivation. The central claim is that CodeContests+ test cases yield higher TPR/TNR on 1.72 million authentic CodeContests submissions (Section 4.4) and better LiveCodeBench pass@1 under RL training (Section 4.5). Neither quantity is a fitted parameter of the model being claimed: the generator, validator, and checker agents are not optimized against TPR/TNR targets, and the evaluation submissions are external contest records with existing pass/fail labels. The use of DAPO [18] is a self-citation (shared ByteDance Seed group), but DAPO is adopted as a standard RL objective and the quality comparison would not change if another PPO-style algorithm were used; therefore the self-citation is not load-bearing. The main legitimate concern is a measurement confound: the intervention simultaneously replaces test inputs and adds custom checkers, so the TPR improvement in Section 4.4 is not cleanly attributable to test-input quality alone, and the CodeContests+HQ subset is selected on the same TPR/TNR thresholds used to argue quality. These are validity threats, not circularity: the reported outcome is an empirical measurement, not the input to the construction. The acknowledged limitation that LLM-written validators can still miss constraints (Section 3.2) similarly weakens the 'verified' label but does not make any claim equivalent to its own premise. I therefore find no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the reliability of the LLM-written validators and checkers and on the accuracy of the original submission labels. The quality measurement uses hand-set thresholds and sample sizes. No new physical or mathematical entities are introduced.

free parameters (2)
  • HQ quality threshold = TPR >= 0.9 and TNR >= 0.9
    Hand-set threshold used to select the CodeContests+HQ subset for RL training; it is an author choice, not derived from first principles.
  • Sample size per problem for TPR/TNR evaluation = 100 positive and 100 negative submissions, capped by availability
    Chosen by the authors to bound compute (300M executions). This sample size affects the statistical precision of per-problem TPR/TNR estimates.
assumptions (4)
  • domain assumption The 'correct'/'incorrect' labels in CodeContests submission data are accurate.
    The TPR/TNR evaluation in Section 4.4 interprets a low TPR as indicating test case flaws, which is only valid if the positive submission labels are genuine. Appendix D.3 shows an instance where the official label is wrong.
  • domain assumption The LLM-written validator programs faithfully capture all input constraints.
    Section 3.2 states that validator agents can overlook constraints, leaving 'a small number of incorrect data' possible, yet the dataset is described as 'verified'.
  • domain assumption A correct solution will pass all valid test cases for a problem.
    The TPR metric assumes that any failure of a correct solution implies a faulty test case. This is standard for competitive programming but depends on the test cases themselves being valid instances of the stated problem.
  • domain assumption LiveCodeBench pass@1 with 15 responses is a stable measure of model performance.
    The RL comparison in Section 4.5 reports a single avg@15 value per condition, with no repeated runs or statistical significance testing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeContests+: High-Quality Test Case Generation for Competitive Programming." pith.science (2026). https://pith.science/paper/GCZAP76H

@misc{pith2026250605817,
  author       = {Pith},
  title        = {Pith review of: CodeContests+: High-Quality Test Case Generation for Competitive Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GCZAP76H}},
  note         = {Machine review of arXiv:2506.05817}
}
read the original abstract

Competitive programming, due to its high reasoning difficulty and precise correctness feedback, has become a key task for both training and evaluating the reasoning capabilities of large language models (LLMs). However, while a large amount of public problem data, such as problem statements and solutions, is available, the test cases of these problems are often difficult to obtain. Therefore, test case generation is a necessary task for building large-scale datasets, and the quality of the test cases directly determines the accuracy of the evaluation. In this paper, we introduce an LLM-based agent system that creates high-quality test cases for competitive programming problems. We apply this system to the CodeContests dataset and propose a new version with improved test cases, named CodeContests+. We evaluated the quality of test cases in CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels to examine the accuracy of these test cases in evaluation. The results indicated that CodeContests+ achieves significantly higher accuracy than CodeContests, particularly with a notably higher True Positive Rate (TPR). Subsequently, our experiments in LLM Reinforcement Learning (RL) further confirmed that improvements in test case quality yield considerable advantages for RL.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A target-blind agent-generated test suite with a consensus-and-validator certification chain found 906 verified accepted-but-buggy submissions on AtCoder and led all baselines on post-cutoff Codeforces problems.

  2. Efficiency of turbulence

    physics.flu-dyn 2025-08 unverdicted novelty 6.0 of 10

    The efficiency of turbulence, the fraction of input energy stored in the flow, appears bounded and may saturate in a power-law manner across several turbulent flows.

  3. MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    A closed-loop test-driven framework - filter tests by LLM self-consistency, refine code in parallel, and select the winner by bipartite code-test mutual scoring - reaches 96.34% Pass@1 on HumanEval without human-writt...

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages · cited by 3 Pith papers

  1. [1]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URLhttps://arxiv.org/abs/2108.07732

    arXiv 2021

  2. [2]

    Seed-thinking-v1.5: Advancing superb reasoning models with reinforcement learning, 2025

    Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng Chen, Hongmin Chen, Riwei Chen, Liangqiang Chen, Zixin Chen, Jinsong Chen, Siyan Chen, Kaiyuan Che...

    arXiv 2025

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  4. [4]

    Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Liu, Jerry Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guanghan Ning, Z. Y. Peng, Kai Shen, Jiahao Su, Jing Su, Tao Sun, Yifan Sun, Yunzhe Tao, Guoyin Wang, S...

    arXiv 2024

  5. [5]

    Competitive programming with large reasoning models, 2025

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, 11 Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. C...

    arXiv 2025

  6. [6]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen...

    arXiv 2025

  7. [7]

    Measuring coding challenge com- petence with APPS

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge com- petence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Trackon Datasets and Benchmarks 1, NeurI...

    work page 2021

  8. [8]

    LiveCodeBench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Twelfth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025....

    work page 2025

Show all 19 references
  1. [9]

    A survey of task allocation and load balancing in distributed systems.IEEE Trans

    Yichuan Jiang. A survey of task allocation and load balancing in distributed systems.IEEE Trans. Parallel Distributed Syst., 27(2):585–599, 2016. doi: 10.1109/TPDS.2015.2407900. URLhttps://doi.org/10.1109/TPDS. 2015.2407900

    2016

  2. [10]

    TACO: topics in algorithmic code generation dataset.CoRR, abs/2312.14852, 2023

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. TACO: topics in algorithmic code generation dataset.CoRR, abs/2312.14852, 2023. doi: 10.48550/ARXIV.2312.14852. URLhttps://doi.org/10.48550/arXiv.2312.14852

    2023 doi arXiv

  3. [11]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    2022

  4. [12]

    Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edito...

    2023

  5. [13]

    Testlib, 2005

    Mike Mirzayanov. Testlib, 2005. URLhttps://github.com/MikeMirzayanov/testlib

    2005

  6. [14]

    Codeforces as an educational platform for learning programming in digitalization.Olympiads in Informatics, 14(133-142):14, 2020

    Mike Mirzayanov, Oksana Pavlova, Pavel MAVRIN, Roman Melnikov, Andrew Plotnikov, Vladimir Parfenov, and Andrew Stankevich. Codeforces as an educational platform for learning programming in digitalization.Olympiads in Informatics, 14(133-142):14, 2020

    2020

  7. [15]

    Can language models solve olympiad program- ming?, 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad program- ming?, 2024. URLhttps://arxiv.org/abs/2404.10952

    2024 arXiv

  8. [16]

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution-based code generation using deep reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id= 0XBuaxqEcG

    2023

  9. [17]

    Qwen2.5 technical report, 2025

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2025 arXiv

  10. [18]

    If multiple feasible solutions exist, output any one of them

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  11. [19]

    testlib . h

    while judging pods are scheduled using a message queue. All test cases are preprocessed and synchronized across judging pods via a network file system, which eliminates the time overhead of compiling and running generators. Execution pods are equipped with runtime environments...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.