{"id":"8049ff8a-e974-4739-8575-3f8f45a45868","arxiv_id":"2508.13948","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"POML is a markup language that structures LLM prompts, embeds multimodal data, and decouples formatting via stylesheets, with case studies showing strong prompt format sensitivity.","lead":"This paper introduces POML, a markup language for writing prompts to large language models, with reusable components, built-in ways to embed documents, tables, and images, a CSS-like styling system, and IDE tooling. It validates the language with a prototype app, a table-question-answering experiment, and a seven-person user study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TableQA case study does not validate POML's central accuracy/styling claims: no baseline, no held-out style selection, and extreme min/max accuracies are reported without confidence intervals.","rationale":"The reader's verdict is CONDITIONAL largely because the TableQA study does not show POML improves accuracy over baselines; that is also the core of my concern. I therefore agree with the reader's overall rationale, and my verdict remains CONDITIONAL/UNCHANGED. However, the reader's formal 'weakest_assumption' focuses on the lossless decoupling of content and presentation. That assumption is not actually central to the paper's claims: POML's own experiments show that what it calls styling changes can affect model performance, and the design goal DG3 is about source-level separation, not semantic invariance. The more precise and damaging issue is the evaluation design in §7.2: the accuracy evidence that is supposed to validate the styling system lacks a baseline, relies on extremes from a random sample, and selects model-specific best styles on the same small validation set without holdout. If that evidence collapses, the central claim that POML's styling system is an effective answer to format sensitivity loses its empirical backing, even though the language artifact itself may still be useful. My concrete test directly targets this gap. No procedural or personal criticism is intended; the issue is the evidentiary link between the experiment and the headline claims.","tokens_in":41196,"tokens_out":12307,"duration_ms":137249,"concrete_test":"Re-run the WikiTQ evaluation (same 283 samples, same 100 styles) adding two baselines: (1) the base POML prompt rendered with its default/central style, and (2) a plain-text prompt with identical content. For each model, report bootstrap 95% CIs for the accuracy of every style and for both baselines. Then perform a 50/50 split: select the best style on one half and evaluate it on the other, repeating 100 times, and compare this held-out accuracy to the baselines. Also recompute the self-correlation of Table 2 excluding styles with zero correct answers on either half. If the POML-selected best style is not significantly more accurate than the default style or plain-text baseline on held-out data, the accuracy-related claims in the abstract and §7.2 are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in §7.2. The TableQA experiment is presented as evidence that POML's styling system improves accuracy or reduces formatting sensitivity, but the design cannot support that. It samples 100 styles from a 74k space and reports min/max accuracy on a 283-example validation set, with no baseline prompt for comparison (neither a default POML style nor a plain-text equivalent). The relative improvements (up to +4450%) are extremes selected from 100 random draws; for Phi-3 Medium the minimum accuracy (0.7%) corresponds to about 2 correct answers, so the denominator is statistically indistinguishable from zero. Table 1's 'optimal style' for each model is chosen on this same small validation set, with no held-out evaluation; the self-correlation metric (Table 2) is computed on the same 283 samples and can be inflated by tied near-zero scores. Consequently the paper demonstrates the well-known phenomenon of prompt-format sensitivity, but not that POML's styling layer, its decoupling, or its tooling improves accuracy or reduces sensitivity over any existing approach. The unique value proposition of the CSS-like styling system therefore rests on unsupported empirical ground.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces POML (Prompt Orchestration Markup Language), an HTML-like markup language for authoring LLM prompts. POML provides structural components (<role>, <task>, <example>), data components for documents/tables/images, a CSS-inspired styling system that decouples content from presentation, and a templating engine. The authors also describe an IDE extension (VSCode IntelliSense, live preview, diagnostics) and Python/Node SDKs. Validation consists of two case studies: PomLink, an iOS agent prototype built in two days using POML, and TableQA, a study of 100 randomly sampled prompt styles applied to 283 WikiTQ samples across 8 LLMs. In addition, a user study with 7 participants assesses usability across five tasks. The paper claims that POML reduces formatting sensitivity and improves accuracy performance, and that its tooling improves developer workflow.","tokens_in":41485,"tokens_out":2941,"duration_ms":34309,"significance":"If the empirical claims were fully supported, POML would be a meaningful contribution to prompt-engineering tooling: the design is comprehensive, the implementation appears substantial (14.8k lines, 37 components, 115 test cases), and the detailed appendices (style search space, three-pass rendering architecture, IR specifications) are valuable. The PomLink case study and the qualitative user-study feedback suggest that POML's data components and live preview are genuinely useful in practice. However, the central quantitative claim—that POML's styling system improves accuracy or mitigates format sensitivity—is not established by the TableQA experiment as presented. The experiment shows, at best, that LLM performance is sensitive to prompt formatting, a known phenomenon, and that POML can generate many formatting variants. The paper overstates this as POML's impact on accuracy. The user study, while informative, is small and lacks a comparison baseline. With targeted revisions to the evaluation and a more measured interpretation, the paper could be a solid systems/HCI contribution.","major_comments":[{"comment":"The TableQA experiment has no baseline. The reported 'accuracy improvements' of 929% and 4450% are relative differences between the worst and best of 100 randomly sampled styles, not improvements of POML over plain-text prompts or existing markup tools. The abstract and contribution (3) claim POML 'demonstrates its impact on accuracy performance,' but the design cannot support that. To support such a claim, the paper must include a baseline condition (e.g., the same TableQA task written as plain text or in an existing format such as ChatML/PromptML) and compare POML-rendered prompts against it. Without this, the paper should be reworded to claim only that styling variations affect LLM accuracy, not that POML improves it.","section":"§7.2, Table 2"},{"comment":"The 'optimal' styles in Table 1 are selected post hoc on the same 283 samples used to compute all accuracy numbers. There is no held-out evaluation of the selected styles. This matters because the PomLink 'Auto' option (§7.1) consumes these fitted results to choose table formats for new prompts. The self-correlation metric (Table 2) is a split-half stability measure, not a validation of the top style. The authors should split the data, select styles on a training subset, and evaluate on a held-out subset; they should also report whether the 'Auto' choice outperforms a fixed default or random style on unseen data.","section":"§7.2, Table 1 and §7.1 'Auto' option"},{"comment":"The extreme relative improvements are statistically fragile. Phi-3 Medium's minimum accuracy is 0.007, which on 283 samples corresponds to roughly 2 correct answers, so the denominator is indistinguishable from zero. Reporting a 4450% improvement from such a baseline is misleading. The paper should report absolute differences with confidence intervals (e.g., Wilson intervals) and should avoid presenting min/max extremes from 100 random draws as evidence of POML's benefit. Similarly, Table 6 performs many Mann-Whitney U tests without multiple-comparison correction; the p-values should be adjusted (e.g., Benjamini-Hochberg) and effect sizes reported.","section":"Table 2, Phi-3 Medium row"},{"comment":"The user study's central claims are limited by the acknowledged small sample (N=7), the restricted 90-minute sessions, and the authors' own admission that most participants 'did not deeply engage with complex stylesheets' (§9.2). The paper nevertheless describes the study as a 'formal user study' and uses it to support POML's effectiveness. I would ask the authors to either temper the language (e.g., 'pilot study') or add complementary evidence, such as a comparison task with plain-text prompts or with another markup tool. The current design cannot distinguish POML's effect from the Hawthorne effect of a new tool in a lab setting.","section":"§8 and §9.2"},{"comment":"The design goal DG3 assumes that prompt presentation can be decoupled from content without changing the underlying meaning the model sees. The paper operationalizes this through stylesheets, but does not verify content invariance: changing syntax, caption styles, or list styles also changes tokenization, ordering, and emphasis, which could alter semantics. This is not necessarily a fatal flaw, but it is a load-bearing assumption for the claim that POML 'reduces formatting sensitivity' rather than simply shifting it. The TableQA study could partially address this by comparing styles that differ only in presentation while holding the exact words constant, but the current analysis does not isolate this. I recommend adding an explicit test or at least a careful discussion of this limitation.","section":"§4.3, DG3"}],"minor_comments":[{"comment":"The term 'Self-corr.' is used without a definition in the main text. It is defined only later in Appendix E. Please define it in the caption or in §7.2.","section":"Table 2 caption"},{"comment":"In Figure 6, the diagnostic message shows 'captionStyle should be one of header, bold, plain, hidden, not \"head\"' but the list of valid values is inconsistent with the text in §4.3, which mentions 'header, bold, plain, hidden'. This is a small documentation inconsistency that could confuse users.","section":"Figure 6"},{"comment":"The style search space diagram (Figure 10) is dense and hard to read, especially the labels for options (a) through (e). A table listing the discrete choices with their cardinalities would improve reproducibility and clarity.","section":"Figure 10"},{"comment":"The paper says 'we randomly sampled 100 styles without replacement for evaluation' but does not report the random seed or whether the same styles were used across all models. Please clarify whether the 100 styles are identical across models and provide the seed for reproducibility.","section":"§7.2"},{"comment":"Table 6 uses arrows (D, ✗, –) with p-values in cells. The notation is not intuitive; a legend would help. Also, the interaction-effect rows are valuable but are not discussed in the main text.","section":"Appendix E, Table 6"},{"comment":"The paper uses the phrase 'rigorous empirical evaluation' (e.g., §7 and §10). Given the limitations of the TableQA and user studies, I recommend a more measured description such as 'initial empirical exploration' or 'case-study-based evaluation.'","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's core contribution—a language and toolchain for structured prompt authoring—is potentially interesting and the implementation appears real. The main problem is that the headline quantitative claim (POML improves accuracy / reduces format sensitivity) is not supported by the evidence as presented. The authors should either add the missing baselines and held-out validation or substantially reframe the contribution as a systems/tooling paper with an exploratory format-sensitivity study. The novelty relative to existing prompt markup languages (PromptML, ChatML, prompt-tsx, SAMMO) is incremental but present in the integration of data components, styling, and IDE support. I do not see grounds for rejection if the authors are willing to make the required empirical and framing changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: POML is a real, thoughtfully designed artifact—a markup language for prompts with data components, a CSS-like styling layer, templating, SDKs, and a VSCode extension. The paper is worth reading and, after revision, citing. But the empirical case is oversold: the TableQA study demonstrates prompt-format sensitivity, not that POML improves accuracy over any baseline.\n\nWhat's genuinely new: the combination is not in prior work. PromptML, ChatML, PromptTSX, and the others cover pieces, but POML puts them together in one language-agnostic package with seven data components and a real IDE extension. The documentation is thorough—37 components, 283 attributes, a three-pass rendering architecture, LSP server. The PomLink case study shows the tool being used for a real iOS agent prototype, and the user study, while small, is honest about usability trade-offs. The authors also list limitations up front (accessibility, small sample, no open-source yet), which is good practice.\n\nWhere the paper wobbles: the TableQA experiment, which the abstract leans on for \"accuracy performance.\" The stress-test note is right. There is no baseline, neither plain text nor an existing markup tool. The best styles in Table 1 are selected post hoc from the same 100 sampled styles on the same 283 validation examples, with no held-out check. The reported min/max accuracies have no confidence intervals, and Phi-3 Medium's 0.7% minimum is about 2 correct answers—statistically indistinguishable from zero. The self-correlation metric is computed on those same 283 samples, so it can be inflated. The paper says this shows \"POML's impact on accuracy performance,\" but what it actually shows is that formatting matters, which was already known. The styling decoupling claim (DG3) is not directly tested either.\n\nThat said, the core artifact seems solid, and the analysis of model-specific style preferences is interesting as a sensitivity study. The user study does not have a control group, but it is exploratory and the authors say so.\n\nWho this is for: anyone working on prompt engineering tooling, structured prompting, or LLM developer experience. It deserves a serious referee, but the referee should push for a fair baseline (e.g., same content as plain text or as a PromptML prompt), held-out style selection, and error bars. With that, the empirical claims can be made to match what the paper actually demonstrates.\n\nRecommendation: send to peer review, expect significant revision on the evaluation section.","headline":"A well-built prompt engineering artifact, but the TableQA study overstates what it proves: it shows format sensitivity, not that POML improves accuracy.","tokens_in":41968,"tokens_out":1744,"would_cite":true,"duration_ms":19169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"POML claims prompt formatting should be a separate, testable layer: on WikiTQ, style changes move GPT-3.5 Turbo accuracy from 6% to 62% and Phi-3 Medium by 4,450%.","keywords":["prompt engineering","POML","markup language","LLM format sensitivity","stylesheet separation","structured prompting","data integration","developer tooling"],"falsifier":"Render the same POML source under two stylesheets and record the exact final strings sent to the model. If two styles that produce identical strings nevertheless lead to different answers, the styling system is not the causal variable; if two styles that are meant to differ only in presentation change which content is visible (e.g., a hidden caption or a collapsed table drops a fact the model needs), the decoupling premise fails. A direct test would compare model accuracy on style pairs whose rendered strings differ only in whitespace or order versus pairs that delete or reorder content.","tokens_in":41104,"feed_emoji":"🎛️","tokens_out":5321,"duration_ms":54323,"temperature":0.7,"pith_summary":"This paper introduces POML, a markup language for writing LLM prompts as structured components (roles, tasks, examples) with dedicated tags for embedding documents, tables, and images. Its central proposal is to separate prompt content from prompt presentation through a CSS-like stylesheet system, so developers can change formatting without rewriting the prompt's logic. The paper argues this matters because LLMs are extremely sensitive to formatting: on a subset of WikiTableQuestions, choosing the best of 100 sampled styles instead of the worst raised GPT-3.5 Turbo's accuracy from 6% to 61.8% and Phi-3 Medium's from 0.7% to 32.2%. POML's value would be that it makes such variation systematically explorable — one 30-line base prompt plus generated stylesheets produced roughly 74,000 distinct prompt styles — and manageable in real applications. A two-day iOS agent prototype and a seven-participant user study support the broader claim that the language and its IDE tooling improve the prompt-engineering workflow.","feed_headline":"Style alone swings LLM accuracy from 6% to 62%","feed_subtitle":"A new markup language separates prompt content from formatting, letting developers test thousands of style variants from one file.","key_machinery":"The load-bearing mechanism is POML's styling system: a CSS-like layer where a stylesheet (JSON or inline stylesheet tag) controls presentation attributes — overall syntax, table serialization format, caption style and visibility, example layout (chat vs introducer vs plain), list markers — independently of semantic markup. A three-pass rendering pipeline (parse into components, build an intermediate representation, serialize through a target writer) makes the decoupling operational, allowing one base prompt to be rendered into tens of thousands of styles and into different output formats (Markdown, JSON, plain text) from the same representation.","core_discovery":"POML's core claim is that prompt presentation can be engineered as a first-class, content-independent layer. Using HTML-like components for intention (role, task, output-format) and data (document, table, image, folder, conversation), and a JSON stylesheet system controlling syntax (Markdown, JSON, XML, HTML), captions, list styles, chat-versus-block layout, and verbosity, a single POML source can be rendered into many presentation variants without touching the underlying content. The empirical heart is the TableQA case study: 100 randomly sampled styles applied to one base prompt on 283 WikiTQ samples produced accuracy swings of 16% to 4,450%, with optimal styles differing by model; some mo","pith_inferences":["An implication the paper leaves implicit is that prompt-styling search can be automated: the stylesheet representation is machine-readable, so an optimizer could sample and score styles the way the paper did manually, and the model-specific rankings suggest the objective function differs per model.","The decoupling premise implies a testable boundary: if a style change ever alters token order, emphasis, or which content is visible, it is no longer pure presentation. That boundary is worth probing because hidden captions and collapsed tables can delete information.","A testable extension is to apply the same three-pass rendering idea beyond single-turn text prompts — for example, to agent system prompts or multimodal contexts, where the content tree stays stable but the target writer changes."],"forward_implications":["If styling is decoupled, a single prompt's content can be re-rendered for different target models, and the TableQA results indicate the optimal rendering is model-specific rather than universal.","Prompt teams can treat formatting as a tunable search dimension: the paper demonstrates 73,926 styles generated from one 30-line base POML file, so systematic experiments can replace hand-maintained parallel copies.","Format-sensitivity findings imply that published prompt-engineering results should specify or control presentation, since a style change can swing accuracy more than many content changes.","The PomLink 'Auto' table format shows the same data can be serialized differently per backend model, making style adaptation a runtime deployment decision rather than a manual rewrite.","Because POML files are text and modular, they fit standard version-control workflows, easing collaboration on prompt changes."],"supporting_citations":[{"why":"Supplies the WikiTableQuestions dataset and evaluation tools used to measure TableQA accuracy across styles.","marker":"[64]"},{"why":"Documents LLM sensitivity to spurious formatting features; motivates the styling system.","marker":"[76]"},{"why":"Documents the 'butterfly effect' where small prompt changes alter performance; cited as evidence of format sensitivity.","marker":"[73]"},{"why":"Shows evaluation consistency depends on prompt format; cited for sensitivity and evaluation implications.","marker":"[89]"},{"why":"Argues content and format jointly affect LLM performance; motivates decoupling and automated style optimization.","marker":"[48]"},{"why":"Reports that prompt formatting affects LLM performance; used to justify exploring stylistic variations.","marker":"[35]"},{"why":"Provides a systematic assessment of prompt sensitivity; supports the claim that presentation variations matter.","marker":"[106]"},{"why":"Represents a higher-level symbolic prompt optimization approach that POML positions against by offering direct markup and styling.","marker":"[75]"}],"fun_headline_variants":["Prompt styling alone swings LLM accuracy by up to 4450%","One prompt, 100 styles: LLM accuracy varies 4450%","POML markup: separate content from style, boost LLM accuracy","Test thousands of prompt styles from one file: POML","LLM accuracy swings from 16% to 4450% with prompt presentation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that changing a prompt's presentation never changes its meaning: POML assumes stylesheets can alter syntax, captions, and layout while the model still sees the same underlying content.","fun_headline_variants_meta":{"raw":{"variants":["Prompt styling alone swings LLM accuracy by up to 4450%","One prompt, 100 styles: LLM accuracy varies 4450%","POML markup: separate content from style, boost LLM accuracy","Test thousands of prompt styles from one file: POML","LLM accuracy swings from 16% to 4450% with prompt presentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1377,"prompt_tokens":687,"completion_tokens":690,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":431,"tokens_out":690,"duration_ms":6556,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:48:17.974143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the same POML source under two stylesheets and record the exact final strings sent to the model. If two styles that produce identical strings nevertheless lead to different answers, the styling system is not the causal variable; if two styles that are meant to differ only in presentation change which content is visible (e.g., a hidden caption or a collapsed table drops a fact the model needs), the decoupling premise fails. A direct test would compare model accuracy on style pairs whose rendered strings differ only in whitespace or order versus pairs that delete or reorder content.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WikiTableQuestions dataset and evaluation tools used to measure TableQA accuracy across styles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents LLM sensitivity to spurious formatting features; motivates the styling system."}],"review_version":1}