Pith. sign in

REVIEW 3 major objections 5 minor 124 references

Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents a dataset of 57,640 unique transactional prompts from public code, an ontology that turns raw prompt strings into structured records, and an analysis finding a Zipf-like spread across languages, tasks, domains, and…

desk verdict Serious resource paper: a much cleaner transactional-prompt dataset than PromptSet plus a useful ontology, but the quantitative claims rest on an unmeasured extraction success rate. read the letter →

arxiv 2608.12905 v1 pith:UYPLW2CV submitted 2026-08-13 cs.CL

Victoria Basmov , Yoav Goldberg , Reut Tsarfaty This is my paper · ORCID
classification cs.CL
keywords transactionalpromptspromptontologyanalysisGitHubLLMpromptingmultilingualengineeringnaturallanguageinterfaces
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompts are usually treated as ad-hoc text, but this paper argues they are linguistic objects worth systematic study. It collects 57,640 unique transactional prompts — natural-language instructions embedded in real software — by statically resolving message and template strings from Python files that call the chat.completions.create API or LangChain's PromptTemplate. It then defines an ontology with fields for language, task, domain, input/output characteristics, instruction structure, and prompting techniques, and annotates the collection with an LLM pipeline checked against manual judgment. The resulting analysis describes a Zipf-like distribution: English dominates at 84.66% of identifiable prompt text, text-to-text is the prevailing modality, question answering leads the task list, and most prompts combine a core task with supporting constraints. The dataset and a browsing interface are released to make prompts available as first-class research objects.

What carries the argument

The carrying mechanism is the transactional prompt ontology paired with a static-analysis extraction pipeline. The pipeline locates target API calls in Python source, resolves message and template strings by recursively tracking variable assignments and function parameters across files, filters out empty or placeholder-only texts, and deduplicates; a manual check of 1,000 prompts was done before scaling. The ontology then splits each prompt into role messages and instruction blocks, labels each block with one of 42 semantic kinds, marks central and negative instructions, records input and output units with their type, structure, modality, language, and variability, and assigns prompting techniques from a fixed inventory of 12. This transforms raw prompt strings into records with enough structure to support quantitative and linguistic analysis.

What would settle it

Run the same extraction and ontology annotation on a sample of repositories that use other LLM client libraries or non-Python languages; if the distributions of languages, tasks, domains, and instruction types shift materially, then the reported Zipf-like patterns are artifacts of the two-API Python-only selection rather than properties of transactional prompts at large.

Watch

Extended reading notes

Core claim

The central claim is that transactional prompts — prompts written to run repeatedly inside software workflows — form a distinct, structured genre of language use that can be captured in a shared ontology and studied quantitatively. The paper supports this by extracting 57,640 unique prompts from GitHub (36,916 from chat.completions.create and 20,724 from LangChain PromptTemplate), annotating each with a multi-layered ontology covering detected languages and explicit language mentions; coarse and fine task and domain; input context, directions, and question units with variability, type, structure, modality, and language; output units with type, structure, modality, language, and answer paradigm; instruction sequences with 42 semantic kinds, central vs. meta status, and negativity; and 12 prompting techniques. On this structured data it finds, among other results, that 89.25% of prompts are grounded in an input context rather than relying on parametric knowledge, that 81.8% of instruction blocks are meta instructions rather than the central task, that constraints make up 33.3% of all instruction blocks, and that the system-user message pair has become the standard format for transactional prompts.

Load-bearing premise

The load-bearing premise is that the static-analysis pipeline, applied only to Python files that call two specific APIs, resolves a representative sample of transactional prompts from public code.

Editorial extensions

If this is right

  • If prompts are structured objects, then instruction-following benchmarks can be built from naturally occurring constraints rather than synthetic templates.
  • The finding that 89.25% of transactional prompts are context-grounded supports the view that production LLM use is predominantly grounded in supplied context rather than parametric knowledge.
  • The predominance of meta instructions (81.8%) suggests that effective prompt design is mostly about control — format, constraints, roles — rather than about stating the task itself.
  • The Zipf-like distribution across 62 used languages and 151 mentioned languages can inform multilingual prompt engineering and evaluation.
  • A shared ontology makes it possible to compare prompts across repositories, tasks, and time, enabling diachronic studies of prompt evolution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension of the ontology to interactive and agentic prompts — not pursued here — would test whether the system-user standard and the observed constraint density are specific to the transactional register.
  • The gap between 62 used and 151 mentioned languages hints that prompts are a site of language planning; whether mentioned languages actually shape output quality is a testable consequence the paper leaves open.
  • Because the data is a snapshot from two APIs and Python only, the absolute frequencies are less portable than the structural ratios; re-running the pipeline on a different client library would reveal which patterns are API-specific conventions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a large collection of 57.5K unique 'transactional prompts' extracted from public GitHub repositories that invoke either the OpenAI chat.completions.create API or the LangChain PromptTemplate constructor. It proposes a structured ontology that describes prompt languages, tasks, domains, input/output characteristics, instruction semantics, and prompting techniques; applies an LLM-based annotation pipeline to the full corpus; and reports a per-field error analysis on 100 manually reviewed prompts. The authors provide descriptive statistics over the annotated corpus (language, modality, domain, task, grounding, instruction kinds, message structure, prompting techniques) and release the dataset, annotations, and a web interface for exploration.

Significance. If the resource is reliable, it would be the first large-scale structured corpus of transactional prompts and a useful starting point for linguistic, empirical, and engineering studies of prompts in software. The paper ships a substantial artifact: the dataset, the ontology, the annotation prompts, a web UI, and unusually transparent per-field error tables. The ontology and the explicit error analysis are valuable contributions even if some quantitative claims later require revision. The main risk is that the extraction pipeline and the annotation quality limit the validity of the reported distributions; these are correctness concerns that can be addressed with additional measurements and more cautious wording, not fundamental flaws in the resource concept.

major comments (3)
  1. [§2, Appendix A] The extraction pipeline's resolution success rate is not reported. The paper moves from 95,806 starting files and 145,553 objects to 57,640 unique prompts, but it never states how many API call sites were attempted, how many failed to resolve to a concrete prompt text, or which failure modes dominated. The manual verification of 1,000 extracted prompts validates precision of the successful extractions, not recall or representativeness. If a substantial fraction of call sites construct prompts dynamically (runtime f-strings, cross-module concatenation, config-driven templates, function-returned strings), those prompts are either filtered out as placeholder-only or never resolved, systematically over-representing static literal prompts. This bias directly affects the central quantitative claims in Section 5, including grounding proportions, variability labels, instruction-kind frequencies, and language/task/domain distributions. Please report per-stage counts of attempted, resolved, filtered, and deduplicated items, categorize the resolution failures, and discuss how the unresolved subset might differ.
  2. [§4, Table 8] The annotation accuracy is uneven and the error analysis is thin for the number of claims made. Output Type is only 60.4% accurate and Directions Text is 69.4%, while the error analysis is based on 100 data points, a single expert, and no inter-annotator reliability measure; several field-level accuracies rest on 100–150 evaluated units. The paper calls this a 'comprehensive error analysis' but does not provide confidence intervals or an explicit statement of how the low-accuracy fields should be treated by downstream users. Since the released resource includes these fields and the ontology is a central contribution, the low-accuracy fields need either improved annotation, explicit low-confidence flags, or a clearer caveat that quantitative analyses using them are provisional.
  3. [§5.3, footnote 13] The grounding claim (89.25% of prompts are grounded) is vulnerable to the extraction bias acknowledged in footnote 13. The footnote concedes that dynamically injected context is not captured, and Appendix B filters out prompts that consist only of unresolved variables or placeholders. These two effects could jointly distort the grounding estimate in either direction: prompts with dynamic context are removed, while static prompts with no context are retained. Please quantify the fraction of prompts in which context is added dynamically or through unresolved placeholders, and report the grounding proportion under alternative inclusion assumptions, or explicitly mark the 89.25% figure as an upper/lower bound.
minor comments (5)
  1. [§3] The 'Input characteristics' paragraph appears malformed: item (3) ends with 'for 2.' and is followed by a stray '2.', and the opening quote in '(1) overall high-level instructions (“answer the question provided by the user)' is unbalanced. Please rewrite this list so each item is complete and the quotation marks are consistent.
  2. [§5.2] The sentence 'Overall, the dataset includes 39,4875 such instruction blocks' contains an apparent typo: given 57,640 prompts and an average of 6.85 blocks per prompt, the expected total is roughly 394,000. Please verify the number and correct the digit grouping.
  3. [§5.1] The text says 'The following seven highly represented languages' but then lists eight languages (French, Russian, German, Indonesian, Vietnamese, Polish, Italian and Dutch). Please correct the count or the list.
  4. [Figure 12 caption] The caption maps both audio and video to 'AUD'; video should have its own abbreviation (e.g., 'VID'). The same issue appears in the inline key ('video=AUD').
  5. [Appendix J] The instruction-block annotation prompt lists a fixed set of block names, but its own example uses 'task description', which is not in that list. Please align the allowed vocabulary with all examples, or explicitly state that the list is open-ended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a descriptive resource paper with an operationalized extraction pipeline, an expert-defined ontology, and manual/LLM annotation, none of which reduces to its own inputs.

full rationale

The paper makes no fitted-parameter or self-referential derivation. Its central outputs are a corpus, an ontology, LLM-based annotations, and descriptive statistics. The extraction pipeline (Section 2 and Appendix A) operationally defines transactional prompts as text resolved from chat.completions.create or LangChain PromptTemplate call sites; the language, task, and domain distributions are then measured on that operationalized corpus rather than derived from the definitions. The ontology categories (Section 3) are explicitly grounded in prior literature, inherent prompt properties, and manual inspection of samples; the annotation prompts in Appendix J encode those categories, so reporting frequencies of those categories is descriptive measurement, not a prediction from a fitted model. The error analysis (Section 4) is a manual human check against the annotation guidelines, not an LLM self-assessment, so it does not use the annotation model's own outputs as ground truth in a circular way. Selection biases (Python-only files, two specific APIs, unresolved dynamic prompts) are acknowledged in the Limitations section and affect representativeness, not circularity. No equation, fitted constant, or uniqueness theorem is invoked, and there are no load-bearing self-citations. Accordingly, no specific circular step can be exhibited, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the representativeness of the GitHub/API-based collection method and the reliability of LLM-based annotation. Both are stated as assumptions with acknowledged limitations, but neither is independently verified beyond small manual samples.

assumptions (3)
  • domain assumption GitHub repositories invoking chat.completions.create or LangChain PromptTemplate in Python files provide a representative sample of transactional prompts in the wild.
    The extraction is biased towards these two APIs, Python files, and public GitHub projects, as acknowledged in the Limitations section. This limits generalizability to other prompt-writing contexts.
  • domain assumption The LLM-based annotation, after prompt refinement, produces labels accurate enough for quantitative analysis.
    The error analysis is based on 100 data points and shows some fields with accuracy below 70%, which threatens the precision of claims that rely on those fields.
  • domain assumption Static analysis can resolve prompt texts from variable assignments and function calls across files.
    The extraction pipeline resolves variables, but the success rate is not reported, and unresolved or partially resolved prompts may introduce selection bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code." pith.science (2026). https://pith.science/paper/UYPLW2CV

@misc{pith2026260812905,
  author       = {Pith},
  title        = {Pith review of: Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYPLW2CV}},
  note         = {Machine review of arXiv:2608.12905}
}
read the original abstract

The behavior of contemporary generative Large Language Models (LLMs) is directly shaped by prompts, unstructured texts that describe the desired output and model behavior. In this paper we argue that prompts are linguistic objects that merit investigation in their own right. To this end, we collect 57.5K unique samples of prompts from GitHub. Specifically, we focus on transactional prompts: reproducible natural language instructions that are integrated into software. To enable the empirical, quantitative study of prompts, we introduce a structured ontology, capturing the properties of prompts as well as their formal and semantic components. Based on this ontology, we transform prompts from unstructured raw texts into richly structured linguistic objects. Analysis of these structured data reveals significant diversity of usage patterns across languages, domains, tasks, and modalities, in a typical Zipf-like distribution where some clearly prevail and others, more diverse, appear in the long tail. To validate the reliability of the ontology-based annotation of the prompts, we perform a comprehensive error analysis across all fields, providing a detailed assessment of annotation quality. We release the dataset together with a browsing and exploration interface (https://github.com/OnlpLab/transactionalPromptsCollection ).

Figures

Figures reproduced from arXiv: 2608.12905 by the authors.

Figure 1
Figure 1. The Prompt Ontology Underlying the Empirical Analysis and the Structured Collection techniques come from a pre-specified list of 12 tech￾niques (e.g. “use of sections", “structured outputs", see [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Prompt Extraction Flow. This figure illustrates prompt text extraction by tracing variables across the [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. User Interface. The top section features a free-text search field, a filter box displaying currently active [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: User Interface. Paginated prompts view with displayed spans. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Annotation Accuracy Per Field (based on error analysis results) [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Most frequent prompt languages in the dataset (top 14, [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Long-tail languages occurring below 100 times in the data [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Explicit Language Mentions (excluding English). Abbreviation key: Armenian = HY; Bahasa Indonesia [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Input non-text modality combinations Combination Count 0 10 20 30 40 50 image image;text audio audio;text audio;image;text audio;image;text;video [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Output non-text modality combinations [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Main input-output modality combinations [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Long-tail input–output modality combinations (less than 0.1% each). The inner circle indicates the input; [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Top four tasks covering over 48% of the data [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Distribution of the top 10 input types across the top 10 tasks in the collection. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Distribution of prompt text lengths in the dataset [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Number of messages per prompt (for chat.completions.create data only) [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: Prompt role sequences by number of messages. [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: Semantic instruction type frequencies [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Distribution of prompting techniques in the dataset. [PITH_FULL_IMAGE:figures/full_fig_p033_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

124 extracted references · 77 canonical work pages

  1. [1]

    education & instruction - 4182 (8.42%)

  2. [2]

    RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models

    RepoDebug: Repository-level multi-task and multi-language debugging evaluation of large lan- guage models.Preprint, arXiv:2509.04078. Yuetian Mao, Junjie He, and Chunyang Chen. 2025. From prompts to templates: A systematic prompt template analysis for real-world LLMapps.Preprint, arXiv:2504.02052. A. M. Mir, E. Latoskinas, and G. Gousios. 2021. Many- Type...

    work page Pith review arXiv 2025

  3. [3]

    business & commerce - 2790 (5.62%)

  4. [4]

    Shubham Vatsal, Harsh Dubey, and Aditi Singh

    Prompting in the wild: An empirical study of prompt evolution in software repositories.Preprint, arXiv:2412.17298. Shubham Vatsal, Harsh Dubey, and Aditi Singh. 2025. Multilingual prompt engineering in large language models: A survey across NLP tasks. ArXiv, abs/2505.11665. Hugo Villamizar, Jannik Fischbach, Alexander Korn, Andreas Vogelsang, and Daniel M...

    arXiv 2025

  5. [5]

    technology - 2468 (4.97%)

  6. [6]

    The top section features a free-text search field, a filter box displaying currently active filters, and buttons for prompt display and download

    media & entertainment - 2040 (4.11%) Figure 3: User Interface. The top section features a free-text search field, a filter box displaying currently active filters, and buttons for prompt display and download. Below, ontology field boxes list available values alongside dynamically updating counts. The Languages box on the right demonstrates selected values...

  7. [7]

    software development - 3863 (7.78%)

  8. [8]

    creative writing & content creation - 1728 (3.48%)

Show all 124 references
  1. [9]

    healthcare & medical - 2485 (5.00%)

  2. [10]

    arts & culture - 1522 (3.07%)

  3. [11]

    food & beverages - 1302 (2.62%)

  4. [12]

    finance & banking - 1933 (3.89%)

    1933

  5. [13]

    artificial intelligence & machine learning - 1175 (2.37%)

  6. [14]

    human resources - 1607 (3.24%)

  7. [15]

    other - 1054 (2.12%)

  8. [16]

    legal & regulatory - 1052 (2.12%)

  9. [17]

    personal development - 1281 (2.58%)

  10. [18]

    gaming - 1005 (2.02%)

  11. [19]

    digital media - 1054 (2.12%)

  12. [20]

    customer support - 898 (1.81%)

  13. [21]

    retail & consumer goods - 804 (1.62%)

  14. [22]

    research, scholarship & publications - 1031 (2.08%)

  15. [23]

    data management - 776 (1.56%)

  16. [24]

    travel & leisure - 984 (1.98%)

  17. [25]

    marketing & advertising - 645 (1.30%)

  18. [26]

    security & cybersecurity - 624 (1.26%)

  19. [27]

    language services - 800 (1.61%)

  20. [28]

    physical sciences - 515 (1.04%)

  21. [29]

    data analytics - 653 (1.32%)

  22. [30]

    cultural studies - 463 (0.93%)

  23. [31]

    geography & locations - 455 (0.92%)

  24. [32]

    government & policy - 564 (1.14%)

  25. [33]

    computer engineering & architecture - 408 (0.82%)

  26. [34]

    mathematics - 502 (1.01%)

  27. [35]

    design & arts - 366 (0.74%)

  28. [36]

    manufacturing & industry - 304 (0.61%)

  29. [37]

    sports - 449 (0.90%)

  30. [38]

    information retrieval - 248 (0.50%)

  31. [39]

    hospitality & food service - 372 (0.75%)

  32. [40]

    personal services - 241 (0.49%)

  33. [41]

    philosophy - 233 (0.47%)

  34. [42]

    agriculture & ecology - 264 (0.53%)

  35. [43]

    transportation - 219 (0.44%)

  36. [44]

    communication & language - 244 (0.49%)

  37. [45]

    document management - 218 (0.44%)

  38. [46]

    hardware & engineering - 215 (0.43%)

  39. [47]

    religion & spirituality - 220 (0.44%)

  40. [48]

    sustainability & environment - 201 (0.40%)

  41. [49]

    project management - 218 (0.44%)

  42. [50]

    user experience & design - 189 (0.38%)

  43. [51]

    safety - 182 (0.37%)

  44. [52]

    academic services & administration - 214 (0.43%)

  45. [53]

    home & interior design - 157 (0.32%)

  46. [54]

    social communication - 194 (0.39%)

  47. [55]

    social issues & policies - 135 (0.27%)

  48. [56]

    veterinary services - 131 (0.26%)

  49. [57]

    biological sciences - 160 (0.32%)

  50. [58]

    assessment & testing - 119 (0.24%)

  51. [59]

    logistics & supply chain - 139 (0.28%)

  52. [60]

    it operations - 93 (0.19%)

  53. [61]

    community & volunteering - 88 (0.18%)

  54. [62]

    energy management - 123 (0.25%)

  55. [63]

    scientific analysis - 85 (0.17%)

  56. [64]

    recreation & leisure - 109 (0.22%)

  57. [65]

    data management & analysis - 82 (0.17%)

  58. [66]

    quality assurance - 81 (0.16%)

  59. [67]

    public services - 86 (0.17%)

  60. [68]

    environmental science - 68 (0.14%)

  61. [69]

    environmental management - 84 (0.17%)

  62. [70]

    general & miscellaneous - 64 (0.13%)

  63. [71]

    data security and quality - 59 (0.12%)

  64. [72]

    security & defense - 70 (0.14%)

  65. [73]

    process modeling & monitoring - 51 (0.10%)

  66. [74]

    administrative services - 66 (0.13%)

  67. [75]

    languages - 30 (0.06%)

  68. [76]

    historical studies - 20 (0.04%)

  69. [77]

    urban development - 54 (0.11%)

  70. [79]

    research & development - 42 (0.08%)

  71. [82]

    Response Format: Response should be always in cleanjson format — don’t use the wordjson or any extra

    politics - 3 (0.01%) E Instruction Block Kinds Inthissectionweprovidethefulllistof42semantic kinds of instruction blocks used in the ontology: - input context placeholder - constraint/restriction - output content requirement - output format requirement - role specification - i...

  72. [83]

    You need to determine if the prompt uses **{technique_name}** ({details['brief']})

    A list of tasks the prompt is intended for. You need to determine if the prompt uses **{technique_name}** ({details['brief']}). Reason step by step as described below, but output only the final answer. Here are your resoning steps:

  73. [84]

    **Study the detailed description** of **{technique_name}** and possible signals of its usage : {details['detailed']}

  74. [85]

    - Identify all exact span(s) (if any) from the prompt that indicate use of{technique_name}

    **Locate candidate spans** in the prompt that demonstrate use of{technique_name}: - Scan for keywords or structures described below. - Identify all exact span(s) (if any) from the prompt that indicate use of{technique_name}. (The evidence has to demonstrate the use of {techniq...

  75. [86]

    **Validate each found span** (if any): - Confirm it fulfills the criteria for **{technique_name}**

  76. [87]

    **Decide usage**: -` is_used = true` if at least one span was found, otherwise `false`

  77. [88]

    Think and explain your decision before answering

  78. [89]

    reasoning

    Return **only** a JSON object with three fields: - "reasoning": a string (max 100 words) where you briefly explain your decision before answering. Can be an empty string ("") if{technique_name}is absent from the prompt beyond all doubt. - “evidence”: a list of strings, each an...

  79. [90]

    Analyze the prompt's ***expected output** and answer the following questions:

    A list of tasks the prompt is intended for. Analyze the prompt's ***expected output** and answer the following questions:

  80. [91]

    Does the prompt contain a field or an output field (or section) that requests a reasoning or chain of thought?

    "Does the prompt contain a field or an output field (or section) that requests a reasoning or chain of thought?" 2 "If yes - does the answer based on this reasoning come before or after the reasoning? In other words, does the promp ask the model 1) to give an answer and then e...

  81. [92]

    role”, “prompt_text

    A JSON list of message objects, each with “role”, “prompt_text” and "message_id" (for easier matching between input and output)

  82. [93]

    Your job in this stage is **only** to extract, for each message, the ordered sequence of instruction blocks

    A list of tasks for the overall prompt. Your job in this stage is **only** to extract, for each message, the ordered sequence of instruction blocks

  83. [94]

    - If none fit, use`Other(...)`

    **Block names**: -To name the structural blocks, use only terms from this list: {blocks} - If several terms apply, pick the single best fit. - If none fit, use`Other(...)`. - You may have multiple blocks of the same kind

  84. [95]

    is_central

    Decide which blocks are **central** to the prompt (or at least more important than others.) This are the blocks for which "is_central" will be set to'true' in step 3

  85. [96]

    instruction_kind

    **Format** each block as an object with **five** fields: -`"instruction_kind"`: the block name -`"instruction"`: the exact substring from`prompt_text` -`"is_central"`:`true`if this block conveys the core task (contains the central task instruction) else`false`. You should try ...

  86. [97]

    **Order**: preserve the order in which blocks appear in the message

  87. [98]

    instruction_kind

    **Splitting**: - Divide into pieces smaller than sentences if needed. Example: {{"instruction_kind": "role specification", "instruction": "As a helpful assistant", "is_central": false, "is_negative": false, "negative_instructions_explanation": null}}, {{"instruction_kind": "ta...

  88. [99]

    instruction_order

    **Empty**: if no blocks, return`"instruction_order": []`. **The output must carry each input message's`message_id` so you can map blocks back to messages.** Be specific, precise and exhaustive. **When you are done, go over your annotaion once again**. Did you mark all the bloc...

  89. [100]

    Decide if it splits into units; if not, treat as one unit

  90. [101]

    For each unit provide the exact verbatim span or variable name from the prompt text

  91. [102]

    ), formatting hints, and any other non-core text. - Never include meta-instructions or process/style guidance (internal procedures, prioritization, memory/style influence, or

    Return all units as a list of strings (one string per unit, or a single-item list if unsplittable). Important: - Extract **word-for-word** only. Do not paraphrase or summarize. - Keep only the **core elements** — minimal spans from the prompt text defining each part. - If you ...

  92. [103]

    type": "description

    For each context_evidence and question_evidence unit, also specify whether it is a description (a textual explanation of what the element represents) or direct_content (actual query/context text or placeholder such as{context}or{question}). - Use "type": "description" if the e...

  93. [104]

    fixed","varying

    context_variability. **Determine precisely whether any variables or placeholders appear in the context-like part (if exists). Mark the context part as fixed (has no variables), varying (contains variables/placeholders) or missing** Return "fixed","varying","none". If the answe...

  94. [105]

    fixed" or

    question_variability. ***Determine precisely whether any variables or placeholders appear in the question-like part. Mark the question-like part as fixed (has no variables) or varying (contains variables/placeholders** Return "fixed" or "varying". Important! Even when the ques...

  95. [106]

    undefined

    language **What natural human languages are used in this unit of directions, context or question** Provide a list of natural human languages used in the unit. It may be a one-item list if only one language is used. If any of the languages used cannot be identified, use "undefi...

  96. [107]

    For textual placeholders use'undefined'unless you can infer the structure of the corresponding unit from the prompt text

    structure (only for context and question units.) **What is the question structure of the unit?** - Single item - Pair of items (type, typeB) - Tuple (typeA, typeB, ..., typeN) - List of items (list of type A) -Dictionaryofitems(key1:typeA,key2:typeB,...) (apairisbasicallyatupl...

  97. [108]

    *_language

    For context units also provide context modality: **What is the modality of the context unit?** - text - audio - image - video For direct text - identify the modality by looking at the text. For textual placeholders use'undefined'unless you can infer the modality of the corresp...

  98. [109]

    Output Identification Identify which spans in the prompt text specify or describe the **output** expected from the model (as opposed to input and other things). This may include different specifications of the output format, style, content etc., output descriptions, output pre...

  99. [110]

    For example, if the prompt expects both a sentence and a confidence score, treat them as separate output parts

    Output Segmentation Determine whether the expected output can be naturally divided into two or more distinct parts. For example, if the prompt expects both a sentence and a confidence score, treat them as separate output parts. If there is only one unified output, treat it as ...

  100. [111]

    undefined

    For each output part, provide the following: a. Output Modality Identify the modality of the output: - text - audio - image - video If the modality is unclear, return "undefined". b. Output Description Either extract the relevant span(s) from the prompt that describe the expec...

  101. [112]

    output": [ {

    Single Output: { "output": [ { "modality": "text", "description": "three follow-up questions that a teacher could ask after reading the student's answer", "description_source": "extracted", "output_language": ["english"], "structure": "list of items" } ] }

  102. [113]

    output": [ {

    Multi-Part Output: { "output": [ { "modality": "text", "description": "customer contact details as a JSON object with keys first_name in Japanese, last_name in Japanese, phone", "description_source": "extracted", "output_language": ["japanese","undefined"], "structure": "dicti...

  103. [114]

    The population of the city{city}

    **Single numeric unit** •Modality: text •Description: "The population of the city{city}" •Structure: single item →**Output**:`numeric`

  104. [115]

    three possible titles for the movie:{plot}

    **List of titles** (we use a singular form for multiple units of the *same* type) •Modality: text •Description: "three possible titles for the movie:{plot}" •Structure: list of items →**Output**:`short text`

  105. [116]

    city names and their populations:{cities}

    **Pair of values** •Modality: text •Description: "city names and their populations:{cities}" •Structure: pair of items →**Output**:`complex (short text and numeric)`

  106. [117]

    diagram of the network architecture

    **Image diagram** (we use a singular form for multiple units of the *same* type) •Modality: image •Description: "diagram of the network architecture" •Structure: single item →**Output**:`image`

  107. [118]

    timestamps:{start}to{end}

    **Timestamp range** •Modality: text •Description: "timestamps:{start}to{end}" •Structure: single item →**Output**:`Time / Date`

  108. [119]

    a text, composed in the format of a one-sentence title followed by an email body

    **A title and a body** •Modality: text •Description:"a text, composed in the format of a one-sentence title followed by an email body", •Structure: "pair of items" →**Output**:`complex (sentence, email)` """ #answer paradigm answer_paradigm_system_prompt = """ You are an exper...

  109. [120]

    medical QA

    Identify the NLP/AI Task(s): •Match the prompt to established NLP or AI task names (e.g., summarization, question answering, NLI, paraphrasing, simplification, text-generation, code-generation, code-fixing, planning, etc.). •Use standard and general terms. Avoid overly specifi...

  110. [121]

    summarization

    Provide a Subtask for Each Task: •Give a more granular description of what the task is doing in this case. •For example, for task = "summarization", a possible subtask might be "article summarization". •Every task must have a corresponding subtask

  111. [122]

    undefined

    Determine the Domain(s): •Identify the domain of the prompt (e.g., medical, finance, news, legal, travel, etc.). •Be specific and exhaustive. If unclear or unidentifiable, use "undefined". •If multiple domains apply, list them all. Output: Return a JSON object with exactly two...

  112. [123]

    languages

    “languages”: a list of objects, corresponding to detected human languages in the text. Important! Make sure this is indeed a natural human language (like German, English etc.) and *not a programming language*. Each object must include three subfields: •“language”: the language...

  113. [124]

    explicit_language_mentions

    “explicit_language_mentions”: a list of objects for each place the prompt explicitly names a natural human language. Important! Make sure this is indeed a *natural human language* (like German, English etc.) and *not a programming language* or just a mention unrelated to langu...

  114. [2016]

    pages 731–747

    Probabilistic model for code with decision trees. pages 731–747. Dhia Elhaq Rzig, Dhruba Jyoti Paul, Kaiser Pister, Jordan Henkel, and Foyzul Hassan. 2025. An empirically-grounded tool for automatic prompt lint- ing and repair: A case study on bias, vulnerabil- ity, and optimi...

    2025 arXiv

  115. [2024]

    a helpful assistant

    Paraphrase types elicit prompt engineering capabilities. ArXiv, abs/2406.19898. WenhaoWangandYiYang.2024. VidProM:Amillion- scale real prompt-gallery dataset for text-to-video diffusion models.Preprint, arXiv:2403.06098. Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Ya...

    2024

  116. [2025]

    Preprint, arXiv:2503.06573

    WildIFEval: Instruction following in the wild. Preprint, arXiv:2503.06573. Jingjing Liu, Zeming Liu, Zihao Cheng, Mengliang He, Xiaoming Shi, Yuhang Guo, Xiangrong Zhu, Yuanfang Guo, Yunhong Wang, and Haifeng Wang

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.