Pith. sign in

REVIEW 4 major objections 7 minor 20 references

Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that code LLMs lose more than 10% when a problem detail is changed counterfactually, and that fine-tuning on such counterfactuals recovers the loss and improves general code benchmarks.

desk verdict Read the CTF-Code benchmark claim with caution; the counterfactual fine-tuning result is the stronger, more credible contribution. read the letter →

arxiv 2505.14597 v1 pith:5OFBTEW4 submitted 2025-05-20 cs.CL

Xianzhen Luo , Qingfu Zhu , Zhiming Zhang , Mingzheng Xu , Tianhao Cheng , Yixuan Wang , Zheng Chu , Shijie Xuyang
show 3 more authors
Zhiyuan Ma YuanTao Fan Wanxiang Che
This is my paper · ORCID
classification cs.CL
keywords codesensitivitycounterfactualperturbationgenerationbenchmarkinstructiontuningdetailchangeLLMsdataselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that code-generating AI models have a measurable blind spot: when a problem description is changed in a small but semantically crucial way, many of them keep solving the original problem and fail. To make this visible, it builds a benchmark of paired problems, CTF-Code, where each counterfactual problem differs only slightly in wording but requires a different algorithm. On this benchmark, most LLMs drop by more than 10% in pass rate compared with the original problems. The paper then shows that fine-tuning on data generated by the same counterfactual technique, filtered for diversity, improves both the new benchmark and existing ones such as LiveCodeBench. The message is that sensitivity to details is a distinct, trainable capability that current data mixes ignore.

What carries the argument

The machinery is a counterfactual pair construction governed by an optimization objective: for an original problem P=(Q,T,S), generate P'=(Q',T',S') that maximizes solution difference under a description-similarity constraint, using normalized Levenshtein distance for D_Q and one minus code-embedding cosine similarity for D_S. Test cases inherit the original inputs and use outputs recomputed by S', so the only changed variable is the description. On the training side, the same perturbation idea is applied incrementally to single-dimension data (e.g., Evol-Instruct), and a k-center greedy selection over semantic embeddings fills the missing third dimension.

What would settle it

A human expert panel would rate the difficulty of every original and counterfactual problem in CTF-Code, or an independent difficulty scorer would assign scores. If the performance drop vanishes after comparing only original-counterfactual pairs with equal rated difficulty, the benchmark is measuring difficulty rather than detail sensitivity. Alternatively, highlighting the changed phrase in the prompt should reduce the drop if the deficit is attentional; if it does not, the deficit lies deeper in semantic comprehension.

Watch

Extended reading notes

Core claim

The central claim is that code LLMs exhibit a systematic sensitivity deficit to problem-description details, and that this deficit is both measurable and fixable. In CTF-Code, every counterfactual problem is constructed so that the description distance is minimized (normalized Levenshtein distance ≤ 0.13) while the solution distance is maximized (one minus cosine similarity of code embeddings), with test inputs inherited from the original problems and expected outputs recomputed by the new solution. The evaluation shows many models, including strong commercial ones, suffer over 10% performance drops on the counterfactual versions. CTF-Instruct applies the same perturbation to existing instruction data and selects a diverse subset (k-center greedy on semantic embeddings) to cover difficulty, diversity, and sensitivity simultaneously; fine-tuned models improve on CTF-Code and on external benchmarks including LiveCodeBench, HumanEval+, and BigCodeBench-hard.

Load-bearing premise

The benchmark's claim that the measured drop is about detail sensitivity rests on the assumption that counterfactual problems are neither harder nor less familiar than the originals; if that assumption fails, the drop could be a difficulty or distribution effect rather than a sensitivity effect.

Editorial extensions

If this is right

  • Benchmark builders should add paired counterfactual problems to their suites to measure detail sensitivity rather than only difficulty and diversity.
  • Instruction data curated with difficulty-only or diversity-only augmentation may leave a detail blind spot; incorporating counterfactual pairs acts as a complementary signal.
  • Reasoning-oriented models in the paper show smaller sensitivity drops, suggesting that runtime reasoning or prompting could be combined with fine-tuning to close the gap.
  • The dual-constraint test-case design (same inputs, recomputed outputs) offers a low-cost recipe for building sensitivity probes from any existing code benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The counterfactual training effect is plausibly a form of discriminative data augmentation: it pressures the model to attend to instruction differences, so similar gains might transfer to non-code instruction-following tasks.
  • The ε=0.13 threshold is specific to normalized Levenshtein distance; using learned or semantic similarity metrics could expose which types of textual edits are hardest for models and make the sensitivity axis more controllable.
  • Because the source problems come from algorithm competitions, the benchmark probes competition-style detail changes; extending the same construction to repository-level or multi-file tasks would test whether sensitivity transfers beyond self-contained puzzles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces counterfactual perturbation methods to evaluate and enhance code LLMs' sensitivity to details in problem descriptions. It constructs CTF-Code, a benchmark of 186 paired problems derived from LiveCodeBench-Easy, where each counterfactual variant minimizes textual change while maximizing solution difference, using LLM-sampled variants, human annotation, and a dual-constraint test-case generation (input inheritance, output reconstruction). Evaluation shows many LLMs drop more than 10% in Pass@1 on CTF variants versus originals. The paper then proposes CTF-Instruct, an incremental instruction-tuning framework that generates sensitivity data from existing single-dimension datasets (Evol-Instruct and Oss-Instruct) and selects subsets via k-center-greedy diversity/difficulty selection. Fine-tuning DeepSeek-Coder 6.7B and Qwen2.5-Coder 14B with CTF-Instruct yields gains on CTF-Code and external benchmarks, including HumanEval+, BigCodeBench, LiveCodeBench, and MultiPLE.

Significance. The paper opens a plausible new axis—detail sensitivity—for benchmarking and instruction-tuning code LLMs. If the benchmark cleanly isolates the changed detail, the reported >10% drops on simple problems would be a useful signal for the field. The training contribution is potentially practical: external benchmark gains (e.g., LiveCodeBench +11.6%) are reported, and the method is validated on multiple base models and data seeds. Strengths include the use of external benchmarks to avoid circular evaluation, human annotation for the benchmark, input-space inheritance in test-case construction, and controlled data-volume/selection comparisons. The central caveat is whether CTF-Code measures sensitivity rather than memorization asymmetry or difficulty shift; this determines the strength of the headline claim.

major comments (4)
  1. [§5.1, Figure 3] The headline claim that LLMs have a >10% 'blind spot' for details conflates sensitivity with task novelty. The original problems are public LiveCodeBench-Easy items that likely appear in training data, while the CTF variants are new. The paper itself acknowledges this in the failure analysis ('This may be due to that the original or similar problems exist in the LLM's training data'). The reported drop is therefore not clean evidence of a general inability to process detail changes. A control condition with same-solution rephrasings (the 'Robust' variants that were explicitly discarded) or novel original problems of matched difficulty would be needed to separate memorization from sensitivity. Without such a control, Figure 3 should be presented as a comparison against public familiar problems rather than a pure measure of detail sensitivity.
  2. [§3.2, difficulty annotation] The claim that only the changed detail explains the performance drop also depends on difficulty being controlled. The annotators judge whether difficulty 'changed too much' (Appendix B.3), but no quantitative difficulty measure is reported for CTF-Code, and the small-size training set (186 problems) amplifies any residual difficulty shift. The paper's own Table 2 reports that for CTF-Instruct data, 5% of pairs differ by ≥0.89 difficulty score, and the construction of CTF-Code has no analogous reported distribution. Since the entire evaluation in Figure 3 is a paired comparison, the authors should report a difficulty-difference distribution for CTF-Code or otherwise quantify that the CTF variants are not, on average, harder than the originals.
  3. [§5.2, Table 3] The main fine-tuning results are based on a single training run per configuration with no variance estimates. The claimed improvements—2.6% on CTF-Code, 4.2% on HumanEval+, 11.6% on LiveCodeBench—appear substantial, but the controlled comparison with existing baselines uses official leaderboard numbers from different settings and the 'w/o select' comparisons are also single-run. Given the known sensitivity of code fine-tuning to seed and data order, the authors should provide error bars or at least multiple seeds for the primary comparisons (at minimum for CTFCoder vs. its baselines on DeepSeek-Coder 6.7B and for the w/o select ablation), otherwise the magnitude of the effect is not reliably established.
  4. [§4.2, Algorithm 1] The k-center greedy selection algorithm contains an inconsistency: the loop variable is written 'for i = 1 to k' but the required amount τ is never used to determine the number of iterations. As written, the algorithm would select k items regardless of τ, and Dsub would not necessarily have size τ. This is a minor algorithmic presentation issue in itself, but it matters because the selection mechanism is a load-bearing component of CTF-Instruct; the pseudocode should be corrected to iterate τ times or otherwise make the size control explicit.
minor comments (7)
  1. [Abstract] The abstract says fine-tuning achieves 'over a 2% improvement on CTF-Code, and more than a 10% performance boost on LiveCodeBench', but the numbers in Table 3 are 2.6% (52.8 vs 48.8? actually 52.8 vs 48.8 is 4.0 on Ori; CTF column 44.5 vs 43.4 is 1.1) and 11.6% on LiveCodeBench; please clarify which comparison the percentages refer to, as the relative vs absolute difference is ambiguous and the CTF-Code gain in Table 3 appears to be about 1.1 points (CTF column) rather than 2.6 points.
  2. [§3.2, Equation 2] Equation 2 combines a code-embedding cosine distance DS and a normalized Levenshtein distance DQ with a scaling factor λ=1.2. The text says λ 'ensures DS and DQ can compute,' which is unclear; a sentence describing how λ was chosen (or that it is a heuristic) would help reproducibility.
  3. [§5.1, Figure 3] Figure 3 lacks error bars and the ordering/selection of models is not explained; also some family labels (e.g., 'Deepseek-R1' and 'OpenAI o1-mini') are plotted at the same x position, making the figure crowded. Error bars or a table with confidence intervals would strengthen the claim of consistent drops.
  4. [§3.2] The threshold ϵ=0.13 is stated empirically but the distribution of DQ values for generated candidates is not shown; reporting the distribution and the number of candidates retained would aid reproducibility.
  5. [Appendix B] The annotation example in Table 6 contains an item with 'Age is hexadecimal' whose effect on the algorithm is not explained; a short note would clarify whether such variants pass the difficulty filter.
  6. [§1, Figure 1] In Figure 1, the counterfactual solution description says 'no matter which number is modified, the result remains the same—double the cumulative sum,' but the caption appears to be cut off; please complete the caption or move the full explanation into the main text.
  7. [§4.1] The §4.1 mentions that 102k generated Dsens are 'evaluated on difficulty and diversity' with Table 2, but Table 2 reports percentile values; please clarify how the difficulty scorer and embedding model were applied and what 'difficulty difference < 1' exactly corresponds to.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CTF-Code drop is an empirical measurement, not a construction artifact, and CTF-Instruct gains are independently validated on external benchmarks.

full rationale

No load-bearing circular step is present. The CTF-Code benchmark is constructed by a heuristic generation-selection objective (Section 3.2, Eq. 1-2) that minimizes description distance D_Q and maximizes reference-solution distance D_S, with epsilon=0.13 and lambda=1.2 set before evaluation. The reported performance drop is measured by pass@1 on newly generated test cases, which is not equivalent to the embedding-based D_S used in selection: a model could in principle solve both the original and counterfactual problems and show no drop. Human annotators explicitly reject 'Robust' variants and judge difficulty changes, so the benchmark is not merely a restatement of the optimization objective. The CTF-Instruct training results are validated not only on CTF-Code but also on external benchmarks (HumanEval+, BigCodeBench, LiveCodeBench, MultiPL-E), so the central improvement claim does not reduce to training and evaluating on the same constructed set. Self-citations (Luo et al. 2024a; Zhu et al. 2024b) appear in related-work context and are not load-bearing. The paper itself acknowledges a possible memorization confound in Section 5.1 ('This may be due to that the original or similar problems exist in the LLM's training data'), but that is a validity limitation about interpreting the drop, not a circular derivation. Overall, the derivation chain is self-contained and externally supported.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The paper introduces no physical or mathematical entities. It introduces a conceptual evaluation dimension called sensitivity, but that is a framing device, not a postulated entity with independent falsifiable handles. The main load-bearing choices are the empirical thresholds and the reliance on external scorers and human judgment.

free parameters (5)
  • epsilon (description similarity threshold) = 0.13
    Used in Equation 1 to retain only counterfactual descriptions with normalized Levenshtein distance below 0.13 from the original. Authors state they set it empirically after reviewing samples in Section 3.2.
  • lambda (scaling factor in Equation 2) = 1.2
    Balances solution difference DS against description similarity DQ when selecting the best counterfactual pair. Authors state it is set so the two quantities are computable.
  • CTF-Instruct subset size = 30k for Evol-based, 10k for Oss-based
    Empirically chosen subset size for the k-center greedy selection in Section 4.2.
  • outlier removal tail threshold = not specified
    Authors remove the tail of samples with extremely large semantic distances before selection, but do not state the exact percentile or distance cutoff.
  • number of generated candidates per LLM = 5
    Authors generate five samples from gpt-4o, gpt-4-turbo, and o1-mini each, noting further sampling produced duplicates.
assumptions (7)
  • domain assumption Pass@1 on hidden test cases is a valid measure of code generation correctness.
    Standard in code benchmarks; the paper evaluates all models with this metric.
  • domain assumption Normalized Levenshtein distance and code embedding cosine similarity adequately quantify description similarity and solution difference.
    These functions define the counterfactual optimization in Equation 1.
  • domain assumption Human annotators with ICPC medals can reliably judge solvability, difficulty, and whether a variant is a true counterfactual.
    Benchmark construction relies on their judgments in Section 3.2.
  • domain assumption The easy subset of LiveCodeBench is appropriate because models solve nearly all original problems, minimizing difficulty confounds.
    Table 1 shows 95.6% accuracy by o1-mini on LCB-Easy.
  • domain assumption The CTF-Code benchmark problems are not present in the training data of evaluated models.
    Contamination would inflate the measured sensitivity drop; the paper does not test this directly.
  • domain assumption The external difficulty scorer from Wang et al. provides valid difficulty estimates for assessing difficulty shift.
    Used in Section 4.1 to show that 99% of counterfactuals have difficulty shift less than 1.
  • standard math The k-center greedy algorithm with embedding distance approximates diversity maximization.
    Algorithm 1 is a standard greedy approximation for the k-center problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals." pith.science (2026). https://pith.science/paper/5OFBTEW4

@misc{pith2026250514597,
  author       = {Pith},
  title        = {Pith review of: Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OFBTEW4}},
  note         = {Machine review of arXiv:2505.14597}
}
read the original abstract

Code Sensitivity refers to the ability of Code LLMs to recognize and respond to details changes in problem descriptions. While current code benchmarks and instruction data focus on difficulty and diversity, sensitivity is overlooked. We first introduce the CTF-Code benchmark, constructed using counterfactual perturbations, minimizing input changes while maximizing output changes. The evaluation shows that many LLMs have a more than 10\% performance drop compared to the original problems. To fully utilize sensitivity, CTF-Instruct, an incremental instruction fine-tuning framework, extends on existing data and uses a selection mechanism to meet the three dimensions of difficulty, diversity, and sensitivity. Experiments show that LLMs fine-tuned with CTF-Instruct data achieve over a 2\% improvement on CTF-Code, and more than a 10\% performance boost on LiveCodeBench, validating the feasibility of enhancing LLMs' sensitivity to improve performance.

Figures

Figures reproduced from arXiv: 2505.14597 by the authors.

Figure 1
Figure 1. While diversity and difficulty have been ex [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of CTF-Code benchmark construction. First, original problems are sent to LLMs to sample [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The evaluation results of Code LLMs on CTF-Code. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The change in model performance as sensitiv [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The performance change brought by the selec [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: The performance varies with the amount of [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The data distribution change trace during the CTF-Instruct pipeline. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The distribution of difficulty scores of sensitive data and its original data. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The distribution of the semantic embedding similarity scores of sensitive data and its original data. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: An example of the original problem [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: An example of the robust version of the original problem. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: The first example of the CTF version of the original problem. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: The second example of the CTF version of the original problem. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: The prompt used to generate CTF-Code Problem. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: The prompt used to generate CTF-Instruct data. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

  1. [1]

    Given a string consisting of three letters ’abc’ in any order, can ’abc’ appear after swapping any two characters at most once?

    Read the original problem and briefly explain the meaning of the original problem. As shown in Figure 11, the meaning of the orig- inal problem is: "Given a string consisting of three letters ’abc’ in any order, can ’abc’ appear after swapping any two characters at most once?"

    work page

  2. [2]

    In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024

    Infiagent-dabench: Evaluating agents on data analysis tasks. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.net. Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. 2024. Open- coder: The open cookb...

    arXiv 2024

  3. [3]

    In Proceedings of the 17th International Natural Language Generation Confer- ence, pages 55–69, Tokyo, Japan

    CEval: A benchmark for evaluating counter- factual text generation. In Proceedings of the 17th International Natural Language Generation Confer- ence, pages 55–69, Tokyo, Japan. Association for Computational Linguistics. OpenAI. 2024. Openai o1 system card. Roger S Pressman. 2005. Software engineering: a prac- titioner’s approach. Pressman and Associates....

    arXiv 2024

  4. [4]

    In Proceedings of the 18th Conference of the European Chapter of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1876–1898, St

    CATfOOD: Counterfactual augmented train- ing for improving out-of-domain performance and calibration. In Proceedings of the 18th Conference of the European Chapter of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1876–1898, St. Julian’s, Malta. Association for Com- putational Linguistics. Shiqi Wang, Zheng Li, Haifeng Qian...

    arXiv 2024

  5. [5]

    swapping any two characters

    Opencodeinterpreter: Integrating code gener- ation with execution and refinement. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, Au- gust 11-16, 2024, pages 12834–12859. Association for Computational Linguistics. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yuk...

    arXiv 2024

  6. [7]

    If there are errors in the Sample Input/Output or in the Test Cases , correct them

    Read and understand the newly automatically generated problem. If there are errors in the Sample Input/Output or in the Test Cases , correct them

    work page

  7. [8]

    any substring can be reversed

    In comparison with the original problem, clas- sify the new problem into three types (Bad, Robust, CTF) and explain what changes have been made. • Bad. The new problem has a signifi- cant vulnerability (logical vulnerability or conflict) and can not be a complete problem. • Robust. The new problem has only a different wording from the original ques- tion,...

    work page

  8. [9]

    YES" if it is possible, and

    Determine whether new test cases need to be added to the CTF problem. For example, the annotator should determine whether the range of data of the new problem is fully consistent with the original problem, and whether the input of test cases of the original problem can be directly executed by the CTF problem. For the first CTF problem, there is no need to...

    work page

Show all 20 references
  1. [10]

    Carefully read and comprehend the original problem 's context, conditions, constraints, and requirements

  2. [11]

    **The modification should be slight but cause a substantial change in the solution approach**

    Identify a critical point in the original problem and think about a modification. **The modification should be slight but cause a substantial change in the solution approach**

  3. [12]

    Ask yourself: Would it change data structures or algorithms? Explain the influence before output the counterfactual problem

    Consider the influence of the modification. Ask yourself: Would it change data structures or algorithms? Explain the influence before output the counterfactual problem. If the influence does not impact the solution approach significantly, rethink another critical point to modi...

  4. [13]

    The modified problem must be consistent, clear, and requires a significantly different solution approach

    Modify the original problem based on the most influential point. The modified problem must be consistent, clear, and requires a significantly different solution approach. Update the sample inputs and outputs to match the new problem condition

  5. [14]

    ###Counterfactual Problem

    Output the counterfactual problem, ensuring the following format: - Before the JSON format, include a section marker "###Counterfactual Problem". - After the section marker, provide the counterfactual problem in the same JSON format as the original, including "question_content...

  6. [15]

    The difference will be assessed through evaluated by the Rouge score, indicating the high similarity in wording, sentence structure, and length to the original

    **Minimal Instruction Change**: Achieve the code change with minimal alterations to the instruction. The difference will be assessed through evaluated by the Rouge score, indicating the high similarity in wording, sentence structure, and length to the original

  7. [16]

    Do not make trivial changes like adding or removing a word, changing the order of words, or replacing synonyms

    **No Trival Changes to Instruction**: Ensure the modification to the instruction is semantic-relevant. Do not make trivial changes like adding or removing a word, changing the order of words, or replacing synonyms

  8. [17]

    **Maximal Code Change**: Your adjustments should lead to considerable changes in the output, impacting aspects like algorithms, data structures, data and control flows, or boundary conditions. The difference will be assessed through both the Rouge score and AST score, indicati...

  9. [18]

    Change every aspect of the code, including the function name, variable names

    **Encourage Trival Code Change**: The code output should be significantly different. Change every aspect of the code, including the function name, variable names. ## Format:

  10. [19]

    Your output should be a #Modified_Sample# dict in **JSON format** as the #Original_Sample# is

  11. [20]

    Using **markdown code snippet syntax** in the instruction and the output

  12. [21]

    ## Examples: {seeds} ## Question: - Original_Sample: Figure 16: The prompt used to generate CTF-Instruct data

    Ensure all characters are **properly escaped** in the JSON string. ## Examples: {seeds} ## Question: - Original_Sample: Figure 16: The prompt used to generate CTF-Instruct data

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.